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In July 1989, the members of Project Cooperation, 10 
community colleges that had been invited to serve as demonstration 
sites for research on institutional effectiveness, were brought 
together for the second national colloquium on institutional 
effectiveness through outcomes measures. This report contains the 
presentations mad- at the colloquium and summaries of subsequent 
group discussions and break-out groups' reports. The first 
presentation was "Assessing Institutional Effectiveness," by John 
Harris, which focused on the evaluation of assessment instruments, 
selection vs. criterion testing, assessment in accreditation and 
self-study, assessment and the improvement of educational quality, 
quality improvement in a global context, the Deming Approach to 
quality control, inspection vs. process improvement, continuity, 
training, and leadership. The second presentation was "Outcomes 
Assessment in Action: The Northeast Missouri State University 
Experience," by Terry B. Smith, which covered the creation of an 
assessment program; value-added assessment; stages of the assessment 
experience (i.e., readiness, implementation, acceptance, and 
commitment); .-vnd eight ingredients for success (i.e., large-scale 
scudent assessment, multiple measures, faculty ownership, student 
acceptance, assessment at all appropriate levels, usable and 
understandable data, flexibility, and the use of data to improve not 
punish) . The final presentation was "Student Assessi-.ent Practices in 
the Community College: Current State, Desired State, Getting There 
From Here," by Kay McClenney. McClenney provided a brief overview of 
the current status of assessment in community colleges, propositions 
regarding desired fu*:ure characteristics and directions for 
assessment programs, and key considerations in the development and 
implementation of assessment programs on local campuses. Lists of 
participants and their institutions, and of discussion group members 
are appended. (JMC) 
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PREFACE 



The second national coUoquitim on institutional effectiveness through outcome 
measures was held July 9-11, 1989, in Nashville, Tennessee. Unlike the first 
colloquium held in 1988 in Colimibia, Maryland, this meeting limited participation 
to member colleges of Project Cooperation and representatives of partner cotmdls 
of AACJC and other specific councils of AACJC. This monograph describes that 
meeting. 

But,first,some history isin order. TheNationalCoimdlonStudentDevelop 
(NCSD), a couuicil of AAQC, holds an annual simimer colloquium to address issues 
relevant to student development professionals in two year colleges. The colloquium 
had been held with sponsorship from American College Testing (ACT). In 1987 the 
National Council of instructional Administrators (NCIA), a council of AAQC, was 
charged by the AACJC to prepare a position paper on institutional effectiveness 
through outcome measures (originally titled "value-added education''). Under the 
leadership of Carol Viola, then NCIA president, NCIA began work in this area. 

Viola appointed Wayne Giles to develop research on the topic, and he enlisted the 
aid of ACT. Early on, the benefit of instructional and student development 
professionals working together was evident Because ACT was already working to 
support some of the NCSD activities, the partnership was natural, and Project 
Cooperation was bom. The project continued xmder the leadership of NCIA 
president Roland Chapdelaine, and NCSD president Walter Bumpus. 

Several activities were initiated. Most notable was a national survey conducted 
by ACT, NCIA, and NCSD of the 1200 jtmior, technical, and community colleges in 
AAQC to determine the current state of assessment in commtinity colleges. 
preliminary report of that survey was presented at the 1989 AACJC Annual 
Convention, and the final report will be distributed nationally before the 1990 
Convention. 

Other activities included a series of regional workshops on assessment sponsored 
by ACT, NCIA and NCSD. As small working conferences, these workshops allowed 
teams to investigate assessment and develop plans useful to their colleges. 

Meanwhile, Project Cooperation invited 10 commiimity colleges nationwide to 
participate in the project as demonstration sites. Each coUege would conunit to at 
least three years of involvement, sele.t a research design (choices included 
longitudinal 'Value-added'' models or predictive models), and use ACT instruments 
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in their research. Participating colleges were also to develop supplemental research 
projects and to use participation in Ihe project to develop and implement their own 
outcome measures of institutional effectiveness, tailored to their specific needs. 

As the current NCIA president, I am pleased to present this report of the second 
Project Cooperation Simuner Conference. This conference in Nashville was 
cospoFcSored by NCIA, NCSD, and ACT. 

The 1989 conference was specifically designed as a working meeting for Project 
Cooperation model sites. Each site team worked together to generate plans and 
refine their research model. In addition, it was a time for site representatives to get 
acquainted, to encourage later contacts, to share success, and seek solutions to 
problems. 

The conference was animportantfirst step, a kick-off, for model sites participating 
in Project Cooperation. Through the information presented by the speakers, the 
sharing of resources among participants, and the time away from daily tasks for 
instructional teams, the projects came into focus and working plans were developed. 

Assessment of institutional measures of effectiveness has become a major issue 
for community colleges nationally. Assessment is no longer seen as merely the 
current educition fad, as this year's buzz word. Most educators recognize that 
assessment can be an importantpart of their planning and development, of assuring 
their students, their constituents, and themselves of quality in their institution. 
Whether mandated by political bodies or recognized as part of any iiistitution's 
quality assurance, assessment can lead to many positive outcomes. The critical self- 
examination it brings, the opportimity for faculty and staff to come together to 
discuss issues of quality, the examination of hard data along with qualitative 
researchmethcds can presentanir^titutionapictiureof itself notpreviouslyenj^^^ 
by most institutions. Such jxiformation can indeed be powerful — powerful to 
legislators, to accrediting bodies, to local commimities, to faculty, and, most 
importantly, to the students who come to om- institutions seeking the best we can 
offer. 

The partnership withACTisavitaione. If ACT, in collaboration T^riith community 
colleges, can develop and test student assessment instruments with natiouv^l norms, 
then colleges can add these assessment tools to their own systems of measurement, 
thus giving them broadly based and broadly accepted indicators of success in 
addition to their local measures. 
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The community colleges partidpaling as model sites in Project Cooperation 
represent the wonderful variety of this nation's commimity colleges: single and 
multi-campus, urban and ruraJ, large and small, technical and comprehensive. 
These colleges are: 

Chemeketa Conununity College Midlands Technical College 

Dyersburg State Community College Orangeburg Calhoun Technical College 

Howard Community College Piedmont Technical College 

Macomb Commimity College Scottsdale Community College 

Massachusetts Bay Community College St. Louis Community College District 

Metropolitan Commimity Colleges of Kansas City Technical College of the LowCoimtry 

We wish the model site partidpeiiits great success and, along with their 
colleagues nationally, eagerly await the outcomes of their efforts. 



Betty Duvall 
President, NCAA 
Dean of Instruction 
St. Louis Community College 
Florissant Valley 
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INTRODUCTION 



This report of the 1989 Project Cooperation Summer Conference proceedings 
clearly reflects the working nature of the conference. Most of the participants 
represented institutions that had comndtted, at least tentatively, to serve as 
demonstration sites and implement a model designed to assess student learning 
outcomes. Othersrepresentedoneof the ProjectCooperation partner councils — the 
National Coimcil for Student Development (NCSD) and the National Council of 
Instructional Administrators (NCIA), or they represented the National Council for 
Research and Planning (NCRP). In any case, all participants shared a special interest 
instudent outcomes assessment and how to use these outcomes to assess institutional 
effectiveness. 

Each of the three people invited to address the conference is nationally renowned 
with regard to at least one topic central to the conference. Following the conunents 
of each speaker, the conference participants were divided into smaUer groups that 
met to discuss the topic and accomplish a specific task related to that topic. 

Participants were grouped according to the type of institution they represented: 
multi-campus urban colleges, small colleges, mid-size colleges, and colleges in a 
''suburban-type'' environment. The groupings worked out extremely well, and we 
came to feel that the groupings revealed common institutional concerns and 
interests in many areas. 

John Harris, our first speaker, helped create the camaraderie and sense of 
"cooperative endeavors" so necessary to the success of the conference. Dr. Harris, 
Assistant to the Provost at Samford University, was asked to comment on assessment 
and institutional effectiveness. He covered a wide range of topics and introduced 
a fresh perspective on how organizations, institutions, and businesses behave in 
order to be successful. Karris' remarks and the ensuing group discussions and 
reports made up the first day of the conference. 

Before the conference formally reconvened on the second day, participants had 
the opportunity of attending a continental breakfast and listening to John Roth of 
ACT discuss uses of ASSET. Following this optional session. Dr. Terry Smith, Dean 
of theCoUege and Professor of Political Science at Northeast Missouri S ' ate University, 
gave the day's feature presentation on the theme of value-added assessment. After 
Dr. Smith's conmients, conference participants again were divided into discussion 
groups according to their type of institution. 
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The optional "breakfast session" on the third and final day of the conference 
featured ACTs David Lutz and Lovely "Kate" Ulmer, who discussed GAAP and the 
various sm-veys available from ACT's Evaluation and Survey Services. Kay 
McClenney, president of MO Educational Consultants, then made the last 
presentations of the conference. Dr. McClenney offered a three-part analysis of 
student assessment practices in commimity colleges: the current state, the desired 
state, and bridging the two. 

On the final day the group sessions centered on developing a list of critical 
success factors. The final group activity involved two phases: in phase one the 
participants met, as before, according to institutional type; in phase two, these 
institutional groups met with their designated ACT consultant to develop a list of 
factors critical to the success of their implementation of an assessment process for 
measuring student outcomes and using these to assess institutional effectiveness. 



DAYl 



ASSESSING INSTITUTIONAL EFFECTIVENESS 

JOHN W.HARRIS, JR- 
Assistant to the Provost, Samf ord University 

Introduction 

My informal survey of this group shows that many or most of you are involved 
to some degree with the following activities: 

• Conduct some kind of follow-up study of graduates— not just during 
periods of self-study but nearly every year or every other year. 

• Survey the opinions of area employers regarding your graduates that 
enter the work force. 

• Survey students' opinions of teaching and effectiveness of institutional 
services. 

• Evaluate student competencies, before students leave, in general 
education skill areas like English composition, reading, and math. 

• Regularly assess the competencies of students in programs that lead 
directly into vocations and university transfer programs. 

• Conduct follow-up evaluation of transfer students to see how they 
perf -^nn at the four-year institutionand how quickly they move through 
the completion of the baccalaureate. 

And I've also learned that about half of you are from states that have mandated 
some type of competency assessment before a student leaves your institution. 

Letmeshare >vith you a number of my impressions or suspicions. For one thing, 
I suspect you've probably heard enough about the theory of assessment and the 
reasons for doing it; and I further suspect that you already have considerably more 
information than is, or ever could be, used or approved to initiate an assessment 
program. 

Another impression I have is that, in general, two-year institutions— junior 
colleges, community colleges, and technical institutes— are more managed than the 
traditional colleges and imiversities; that is, on two-year campuses there is more 
rational and systematic management going on. 
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A third impression is that, though a lot more assessment data are available now 
than five years ago, the use of the data in institutional management has not increased 
substantially. For example, you might know the average exit scores of your students 
but not know what they consider the major difficulty on campus. Or, if you look at 
the students' writing skills, it might be difficult to identify the most common error. 
Or, even if you are able to pinpoint the most common errors in English or math, it 
is quite another matter to use this information to reshape cunicultmi or improve 
instruction. 

In my experience, the use of assessment data to reshape and upgrade instructional 
programs is relatively xmcommon, though two-year institutions have made some 
progress. And I bdieve the effective use of assessment data to improve the 
instruction in our colleges and imiversities is a crucial consideration. 

My interest in assessment goes back a long way. I first became interested in 
assessment because I thought we could not systematically, rationally, and 
pragmatically improve teaching tmtil we knew exactly what students were actually 
doing instead of what we thought they were doing or what they were expected to do. 

So my dream has been to try to find ways to use assessment— whether teacher- 
made, outside, or whatever kind — to give us information on how to improve 
instruction. And, though I hate to admit it, I feel that we're a long way from 
accomplishing that dream— even in the community colleges, which seem to me the 
most focused of our institutions in meeting the continuing education and technical 
instruction needs of the commimities they serve. 

Evaluating Assessment Instruments 

Now I'd like to touch on what I consider some of the most important elements 
of an effective assessment program. A very basic component of any assessment 
program is a sound and systematic way to analyze tests. The two-dimensional 
matrix I have shared with you is a simple but effective way of doing tliat. If, for 
example, you are interested in doing an across-the-coUege assessment of students' 
knowledge of history, I strongly urge you first to Ox der a sample copy of the test and 
subject it to a thorough item analysis. This basic matrix system allows you to analyze 
the test items in terms of content (by columns) and of mental functioning (by rows). 

Also, it is very important to analyze test items against curricular expectations, to 
ensure that you are not selecting test items that are not typically covered in the 
curriculum for that subject area. The little book by Morris and Fitz-Gibbon, listed 
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in the section on test -"^lection, is u good resource on how to systematically evaluate 
atestintennsofcou. - objectives. For those of you in volvedin helping faculty select 
tests to go with particular courses, it is an invaluable tool. I also recommend 
Gronlund's Constructing Achievement Tests. 

Selection vs. Criterion Testing 

There is an important distinction in assessment that even people with a 
background in testing do not always consider— the difference between a selection 
test and a criterion test. A very basic dilemma can arise if you use a test designed for 
selection to measure outcomes achievement. If you use test items that everybody 
gets either right or wrong., traditional test tiieory dictates that you throw them out 
because they fail to discriminate. The purpose of a s^ -xrtion test is to discriminate 
among individuals according to how they score on the test So if you have items that 
everybody gets either right or wrong, those items clearly do not discriminate. 
According to this iheory, the best item is one that 50 percent get right and 50 percent 
get wrong. Yet it is possible that, given excellent insfaruction and a large number of 
brightstudents,a very large niunber of students could give correct answers to sound 
items that are measuring what they actually learned. 

If you continue to reject items that everybody— or almost everybody— gets right, 
and at the same time your dean is demanding outcomes assessment, then you can 
very easily put yourself in a no-win situation. 

Let me introduce a key term. When we select test items that focus on creating 
maximum discrimination among individuals, we are not selecting items that are 
treatment sensitive. The following example illustrates an application of the tenn. 

Suppose you're teaching a math fundamentals course effectively and applying 
the latest techniques, while I'm teaching the s:jne course in the old way and not 
doing a very good job at it. Suppose also that we're both using the same test; we both 
must throw out essentially all the items except those that roughly 50 percent of the 
students get right. As a result, the students that emerge as the most skilled, 
regardless of teaching, are those with the most native ability or intelligence, because 
basically we're throwing out the effects of instruction. 

Almost all the research on outcomes of education indicates that there is no 
significant difference in the method of teaching used. How can this finding— which 
sounds absurd— be possible? The answer is simple: Though very different teaching 
methods with very different degrees of effectiveness are in use, the tests developed 



byournationaltestingorgamzations,wWdiare constructed on the basisof selection, 
do not reflect or measure the effects of instruction. This happens because the tests 
keep the items that create maximum discrimination and reject those items that feil 
to discriminate 

Lef s develop this notion a little further. Testing began in this country as a means 
of selecting out of a large pool the most capable individuals (the old Army Alpha 

test,forexample).ThebasicgoalofaUtestshasbeentoa«atemaxiniumdiscrimination 
oranormal curvedistributionin terms of verbalability and quantitativeability. The 
normal curve of distribution — or bell curve— is created by including primarily test 
items Cjat approximately half the test takers get wrong. 

After general ability tests, achievement tests were developed usirig the same 
item selection criteria. Again, if a large group is well taught and as a group scores 
well on a test, those items must be thrown out. What we need are tests that tell us 
whaf s working and whaf s not workmg in our instructional programs, so we can 
make the necessary changes. But what the existing tests tell us is who's smart and 
who's dumb— not who learned the most and who the least from a given course. 

In contrast, the treatment-sensitive approach to test item selection works this way. 
Suppose you're teaching a math course to a dass of relatively tmprepared students, 
and you teach modules one and three very welL You give all students a test before 
you start the course and the same test after the course is over. The items that 
discriminate between those who did not know at the beginning but do now are the 
ones you keep; they reflect the effect of instruction. Conversely, if the performance 
on an item by the group as a whole is the same before and aifter the course, then 
obviously it is not treatment-sensitive and a poor discriminator. 

I urge you to read as much as possible about the difference between tests that are 
designed to select people and tests that are designed to identify people's acquisition 
of a body of knowledge. It is crucial to realize that American testing has essentially 
focused on ranking people against each other rather than comparing people to the 
proportion of a body of knowledge they know. 

Assessment in Acaeditation and Self-Study 

Atpresentlchair theNationalAdvisoryCommitteeforthe Secretary ofEducation. 
Everj' acaediting body in the United States that wishes to be recognized by the 
Secretary must come before this committee— after undergoing intensive staff 
analysis— every five years. It used to be that the focus of the acaeditation review 
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was such resources as number of Ph.D/s, number and type of books in the library, 
and range of instructional resources. It was a resoarce^ven model; we assumed 
that if the above things were in place, good things would come out at the end. 

I assume you all know that a number of studies have convincingly shown that 
there is virtuaUynocorrelationbetweenmeasurediBSoxircesandmeasxire^ 
What we on the National Advisory Committee are trying to do is steer accrediting 
bodies toward dealing with the institutions and programs they represent so that the 
self-studies will contain documentation that institutions have examined student 
outcomes and the implications of those outcomes for program modification. We 
want to encourage them to use assessment as a feedback tool for implementing 
ongoing improvement. 

That pressure is likely to continue. Last week I met with the heads of the 
accrediting bodies of the American Medical Association. Both the medical school 
accreditor and the AMA accreditor said they have relied heavily on outcome 
measures for years. Increasin^y,accreditingbodiesarecomingbeforeiiswith^^ 
evidence that they are looking dosely at outcomes. The trend is spreading and here 
to stay. 

What willbe expected of institutional self-study in theyears ahead appears quite 
dear. If you are involved with assessment in a commimity college or technical 
institute, most likdy the accrediting body will concentrate on and ask you to 
complete the section that relates to institutional effectiveness. The self-study modd 
we recommend involves five dements: (1) A relativdy general statement of 
purpose; every statement of purpose should be linked to a measurable result (We 
prefer to use the term ""result" rather than dther "goal"" or "'objective'" because there 
is no consensus on the distinction between the latter two.) (2) An operational 
statement of a desirable behavior or action that can be observed. (3) Some means of 
assessing or determining whether that desirable outcome is actually taking pkce. 
That is, for each student goal, how will you achieve and assess it? (4) A person who 
is assigned the responsibility of carrying out the assessment. (5) A description of the 
use of the assessment results. What is primarily involved here is actually 
docmnentationthatthetestresults havebeenapplied in an organized andsystematic 
way to reshape weak areas of the curriculum . 

Assessment and Quality Improvement 

I would like to discuss a topic related to point five above, th^ use of numbers — 
statistics— to improve quality in an enterprise. Now we know that indubtries have 
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long been involved with a concept called statistical process control; Ford Motor 
Company, for example, has encouraged community colleges to teach this concep t in 
connection with quality improvement But my overall impression is that the 
educational commxmity has a great deal to leam about the use of numbers to 
improve institutional management 

I am currently conducting a national survey to find out what institutions are 
doing in terms of using management strategies to improve quality. Again, initial 
findings indicate that community colleges are in the forefront^ some specific 
examples being Delaware County Community College in Philadelphia, Fox Valley 
Commimity College in Wisconsin, and West Palm Beach Commimity College, It 
appears likely that you will hear a great deal more about the quality improvement 
approarliiiiconimimitycoUegesthaninfoxir-yearcoUegesandimiv^^ 
time to come. 

There is one basic problem withmost con ventional education statistics which are 
taken from the social sciences. The social sciences tend to take snapshots of reality 
that do not fector in time. In contrast, whenever a physical scientist measures 
something, she always factors timeinto themeasurement of thephysicalphenomenon 
imder study. 

When we take a picture of a student or employer population at a given time, we 
tend to lay it aside and think we have discovered something. Then, at some later 
time, we may examine that population using a different instrument with very 
different questio.os. But it is very important that we not simply isolate and enumerate 
statistics; we most Ihik the two statistical pictures, think about the connections and 
changesthathaveocairred. Inshort, wemustflw^/yze, Qualityimprovementpeople 
want to tiack a social or demographic event over time to see whether things are 
improving or deteriorating. The original measurement is important in establishing 
whether the general movement is up or down. 

The same holds true for surveys of employer opinions. If you conduct such 
surveys, I recommend that you use the same instruments with the same questions 
repeatedly so the chief officers of your institutions can track whether local emp loyer 
opinions are getting more positive or more negative. Without comparability over 
time, it is nearly impossible to meaningfully compare or monitor assessment results. 

A lot of the statistics we have leamed are very complex— parametric and 
nonparametric statistics that are used in advanced research. But Tm talking about 
using ntunbers for the effective management of a school. And, for this purpose, one 
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of the simplest and best tools is the basic line chart. Figure 1, for example, is a line 
chart thit t:acks the placement rates of a school's secretarial graduates over a ten- 
year period; the chart indicates very clearly whether the trend is up or down. 



Kgurel 

A line chart tracking the 
placement rates of a school's 
secretarial graduates 
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(SOURCE: Quality Assurance for Private Career Schools. J. Harris, S. Hillenmeyer, & 
J. V. Foran. Washington, D.C: The Association of Independent Colleges and 
Schools and McGraw-HiD, 1989, p. 54. Reprinted by permission of the authors.) 

Walk through a manufacturing plant — and this holds true for many service 
organizations as well — and you will see charts that show whaf s happening to sales, 
to scrap, to prices — and aUtheinformation can be tracked onacontinxiingbasis.Th^ 
is not true of most colleges. 

Not long ago I visited a state xmiversity with an enrollment of 1 0,000. 1 asked one 
basic question: "How many of the students who begin here graduate in five years?" 
They couldn't tell me. I was astonished. This is not the way you run a business. One 
of the serious problems in this country is attrition, and this school die! not even know 
how many, let alone the reasons why people were not finishing. They lacked critical 
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fundamentalinfonnation— retentionrates. It would seem to me crudal to know that 
if you are in a private college and not graduating 60 percent of the students who 
btgin there, you are below the national average, or that if 1 0 percent of y oiu- school's 
accounting graduates pa«5s all parts of the CPA exam on the first attempt, you have 
a very successful program because the national average is about 3 to 4 percent 

ThebasiclinechartisstiUa very effective toolforrepresentingandtrackingthese 
statistics and trends. A second, more recent type of chart, whidi I first encountered 
while doing quality improvement v/ork in industry, is the Pareto chart. Pareto was 
an Italian who lived before World War H, and to him we owe the 80/20 Rule: 20 
percent of your students (or any group) cause 80 percent of the problems. A Pareto 
chart is nothing but a column chart in which you always use the total nimiber of 
occurrences as the top figure for the left scale and then make that parallel to 100 
percent on the right vertical scale. 

Figure 2 

A Pareto chart showing STUOBiTDissATtSFACTiONS 
students responses to a 
questionnaire about their 



dissatisfactions. 




(SOURCE- Quality AssurancefbrPrioale Career Schools. J. Harris, S. Hillenmeyer, & 
J. V. Foran. Washington, D.C: The Association of Independent Colleges and 
Schools and McGraw-Hill, 1989, p. 55. Reprinted by permission of the authors.) 
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The Pareto chart in Figure 2 shows students' responses to a questionnaire about 
their dissatisfactions- You'll note that the most frequent response is always the 
leftmost column (in this case: "course compacted into too little time'O with the 
remaining responses foUowingin descending order from left to right Such a chart, 
very easy to read and interpret, makes much more sense from a management point 
of view than analysis or variance tables. Though these tables might give you more 
statistical fine-tuning, you do not make administrative decisions on the basis of 
minor variations; you need the big picture. 

Let me give you one example of how a simple survey can help to correct a major 
misperception. At another large university it was commonly believed that a large 
niimberof students weredroppingoutbecauseofiiiadequatestudentaidfu^ So 
my friend decided to do a systematic analysis of why students were jiot finishing. 
He discovered that the major reason had virtually nothing to do with student aid. 
Dropping out was overwhehningly related to inadequate high school preparation. 
So they could have dramatically increased student aid with no likely effect on 
retention. The main point it is one thing to osswrnewhafs making a difference and 
sometimes quite another to know for sure whaf s making a difference. Surveys and 
anal3rtical studies enable you to find out Devices such as line charts and Pareto 
charts help you depict and digest the information in an intelligible way, I am 
convinced that one thing we are still not very good at is arraying easily collectible 
data in ways that make sense, that can effect positive change. We have an 
o verabimdance of data that are not being used. Learning how to use line and Pareto 
charts can help change the picture. 



Figure 3 

A fishbone diagram for a 
manufacturing organization. 




MATERIALS 



MANPOWER 




PROBLEM 



METHODS 



MACHINERY 



(SOURCE: Quality Assurance for Private Career Schools. J. Harris, S. Hillemneyer, & 
J. V. ForaiL Washington, D.C: The Assodation of Independent Colleges and 
Schools and McGraw-Hill, 1989, p. 59. Reprinted by permission of the avithors.) 
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This first step in solving a problem is identifying the causes of the problem. A 
structured way of identifying the causes of a problem is to use the Pareto chart in 
combination with the fishbone diagram. The Pareto chart lets you see the major 
problem, and then you use the fishbone diagram with your work group to identify 
the major cause. Please look at the typical fishbone diagram in Figure 3. In using a 
fishbone diagram, it is important that you not let people plug their pet solution into 
the "problem" box before the problem has been identified with certainty— which is 
often tempting. Once the main problem has been identified, the fishbone diagram 
is the tool for systematically working through all the possible causes of a particular 
problem. 

Lef s look at an example from manufacturing; refer to Figure 3. Suppose yom 
shop makes clamps that secure the hoses through which water passes into the 
various parts of your motor. A lotof the clamps are being badly made and are ending 
up on the scrap heap. So you draw a fishbone diagram and enter "too much scrap" 
in the problem box. The fish's "spine" (the horizontal line leading to the problem 
box) has four diagonal ribs labeled Manpower, Materials, Machinery, and Methods. 
Once the problem is identified, you then determine in which of the "Four Ms" the 
problem basically lies. The Four Ms provide a systematic procedure for qualify 
teams to brainstorm for the most likely causes of the problem, the principal defect. 
If we apply this analogy to education, the counterparts would be faculty and staff 
(manpawerXstudents (materials), instruction (methods), and curriculum (xiiachineiy)— 
as shown in Figure 4. The Pareto chart and fishbone diagram together lead you to 
the major cause of the problem; once you have it pinpointed you can focus on the 
solution. 

STUDENTS FACULTY/ 



Figure 4 

A fishbone diagram applied to 
postsecondary education. 




PROBLEM 



INSTRUCTION CURRICULUM 



(SOURCE: QMlityAssurancefoT Private Career Schools. J. Harris, S. Hillenmeyer, & 
J. V. Foran. Washington, D.C.: The Association of Independent Colleges and 
Schools and McGraw-Hill, 1989, p. 59. Reprinted by permission of the authors.) 
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Quality Improvement and the Global Picture 

The game of inda«<xial competition has become global, and the key trends and 
players are well known by now. The Japanese models of industrial production have 
been well publidzedintWscountry. Thereisnoshortageof examples tosuggestthat 
the Japanese know a lot more than we do about how to improve quality and increase 
efficiency. 

When Mitsubishi came to North Carolina, they wanted all their workers to have 
a knowledge of the basic statistical techniques we have discussed. They had to hire 
Amencangraduate students to achieve that. In Japan, any high school graduate would 
have that knowledge. 

Several years ago an American-owned and -operated TV plant near Chicago had 
something on the order of 31 defects per 100 TV sets produced. A Japanese firm 
bought the plant, kept the same machinery, Uie same workers, the same supervisors, 
the same middle management, and put a few of their people at the top of the 
management ladder; in a short time they reduced the number of defects to about 6 
per 100 TVs. 

For an example dose to home, look at the Nissan plant right here in Tennessee, 
which assembles trucks and cars. Tennessee workers at Nissan often meet or exceed 
the quality of work done by Nissan workers in Japan. It can be done in the U.S.A. 
General Motors is building its Spring Hill Saturn plant in Spring Hill, Tennessee. If c 
one of the largest industrial facilities ever built. If s a high-risk venture for GM; the 
fate of GM could hinge on Saturn's success. If s obvious tha** the world will be 
watching to see if the Tennessee GM plant can equal the quality of the Tennessee 
Nissan plant. 

These examples illustrate why major industrial people and politicians in this 
country are seriously concerned about the position wf the U.S. in the global 
marketplace. President Bush named the president of the Xerox Corporation to head 
''Quality Month"— October 1989. Coming, the insulation company, has started 
working with local school systems in Coming, New York, to teach basic quality 
concepts. 

One of the discouraging things I hear is that many bucinesa people are at the 
point of throwing in the towel and walking away from involvement in higher 
education. Many of them feel they are not getting an appropriate return on tiieir 
investment and have to retrain college graduates once they hire them anyway, /did 
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they believe that a contimiing major defidency of the U.S. in the arena of international 
ipdustrial competition is an ill-prepared workforce. I understand that Motorola, in 
an attempt to revive its fortxmes, opened a plant near Chicago. They wanted to hiie 
about 200 people with tenth-grade level skills but could only find about 60 who 
could meet minimal performance standards. That indicates a major deep-rooted 
problem. 

The Europeanmarket is also becoming more of a force. In 1992 twelve European 
nations will combine forces, and their economic strength will then be massive. The 
European scene is changing dramatically, and Europeans are concemed about our 
bud^t and trade deficits. One student in a class I recently taught was from Korea. 
She told the class that the typical Korean high school student goes to high school six 
days a week from 8:00 imtil 6:00 and studies every evening. Korean high school 
students don' t usually date; in our coimtry you're a social outcast if you're not dating 
by the time you're tWrteen or fourteen. 

All these examples point to an inescapable conclusion — ^the rest of the world is 
working much harder than we are on building a talented manpower base. Buddy 
Karelis, head of the Fund for Improvement of Postsecondjiry Education, asked if I 
could get the Europeans interested in a cross-national assessment of student 
achievement in higher education similar to the cross-national comparison for 
mathematics that was conducted at the secondary level. You may be interested in 
one of the findings of that Jtudy: U.S. high school students scored lower in calculus 
than the students of any other industrial nation except British Columbia, and 
calculus is not taught in high school in British Colimibia. You may think this result 
isduetoourlargenumber of disadvantaged kids. But if you take the top one percent 
of our students and compare them with the top one percent of other cotmtries, ours 
still come out behind. I don't know exactly when or how, but the industrial 
leadership in the U*S. is likely to do something to remedy the situation. They're 
either going to do it uith us or around us. 

There are some promising signs. U.S. Steel has turned things aroimd and is now 
producing greater tonnage of steel with a fraction of the work force they had 
previously. Change is coming, and the name of the game is producing high-quality 
products at the lowest cost. If you can do that, you'll get all the resources you need. 
But those that are content to do business as usual are not going to make it. One thing 
seems clear: U.S. industries are not going to sit still while our economic resources 
are depleted just because we cannot get oiu: educational act together* 
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As I've said, a basic problem is that we have not found the ways to take 
assessmen c data and use them to make the improvements we know should be made. 
Everyone in this room knows at least one program in their institution that is not 
achieving its purpose. It is essential that you act on that knowledge and start 
working immediately to turn the program around. Such action is the foundation of 
improving institutional quality. 

The Deming Approach 

I wonder if you are familiar with the name Deming? Douglas MacArthur 
arranged for Deming to teach techniqu ; of quality control and improvement to 
Japaneseuidustrialistsinthelatel940sandearly 1950s He taught the Japanesebasic 
statistical process control and participatory managencent practices, both of which 
came directly out of the United States. The irony is that the Japanese adopted these 
theories and put them into practice; we did not. So we have both the theory and the 
test technology. What we have not shown so far is the will to use our theory and 
technology to effect significant change. 

Again, I don't want to give the impression that the picture is totally dark. I am 
vexy impressed, for example, with the Baldrige award for quality improvement, 
awarded by the Department of Commerce. The criteria for this award could be 
applied very efifectiv ely, I think, to our colleges and universities. Florida Power and 
light is the first ^ ^itxxcan company to compete for and receive the Deming Prize. 
And, two years ago, the same company gave a large amount of money to the 
University of Miami to work on the teaching of quality improvement concepts. One 
of thebest eftorts in this area is the Ford program in community colleges, whose aim 
is to teach quality improvement principles to their local suppliers and installations. 

In his book Out of the Crisis Deming describes the ills of American industry and 
puts forth a fourteen-point plan for transforming America's production and 
management methods. Because Deming considers the state of American industry 
as very serious, his proposals are drastic and demand a fundamental change in ways 
of thinking and solving problems. In the handout chapter titled "The Value of 
Quality Assurance" I have summarized Deming's fourteen points and explained 
how they could be applied to a two-year college. Although my example is a 
proprietary school, I feel the points could be modified and applied more broadly to 
other types of institutions as well. 

If you don't want to tackle Gui of the Crisis, Mary Walton, a journalist, has written 
a very accessible book on Deming's theories called The Deming Management Method. 
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Though I leave it to you to read and think about Deming's fourteen points and 
how they might be applied to your institutions^ I'd like to conunent on Deming's first 
point 

"Create constanqr of purpose toward improvement of product and service, 
with the aim to become competitive and to stay in busine^, and to provide 
jobs." {Out of the Crisis, 23) 

"Create constacy of purpose." Sounds basic and simple, doesn't it? Though I'm 
neither a business expert nor an economist, I sense that too few U.S. companies see 
their purpose as "staying in business and providing jobs." Rather— and this may 
sound cynical to you— in too many cases the aim is to stay in business just long 
enough for a few special shareholders to make a short-term haul, then to sell quickly 
and move on to another business venture. 

The last time I was in Paris on special assignment, I met a native Calif omian who 
has been in business all his life. He raises venture funds in Europe. He predicts that 
in about two years the U.S. will experience an economic crash that -wtR make the 
October 1987 "Black Monday" look mild. Wlien I asked what would trigger the 
collapse, he said it would be neither the national budget deficit nor trade imbalance, 
but the LBOs— leveraged buyouts. He felt that the greatest threat to the health of 
American industry is posed by unprincipled opportunists, people who are not in 
business primarily to produce a quality produrt but to get in and out of a business 
with the greatest profit in the shortest time. He agn^ed with Deming that staying in 
business and providing jobs should be the priinaiy purpose. And the key to 
succeeding in this piupose is willingness to innovate. 

Now innovation does not mean mechanically adopting every lufty new trend or 
program that comes along. Productive innovation mv^ans figuring out exactly what 
your purpose is, determining to what extent you are failing to achieve that purpose, 
and then finding specific, practical ways of coming closer to the purpose. 

Let me suggest a term that is very useful: alignment. What we in education 
desperately need is to align our practices with our statements of purpose. We are 
inclined to say anything in a statement of purpose— the mort lofty and idealistic, the 
better. Then we promntiy develop a curriculum that has nothing to do with the 
purpose. 

Businesses— atleast the successful businesses— don't operate that way. The new 
areas or activities they consider are always tied to their main purpose or mission. 
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They move ahead but never lose sight of the main purpose that enabled them to 
succeed. 

Here is where assessment can play a key role. Assessment lets you know when 
something' s not working and must be changed. Tm not referring to fundamental 
product research and development; Ym talking about scrutiny and analysis of the 
operation itself. This kind of ongoing self-examination aeates a sense of constant 
improvement. 

I suggest that we put less emphasis on MBO concepts and approaches. Start 
thinking instead in terms of a continual spiral ofimprovermnt Another way to express 
it is incremental improvement. We are not talking about a revolution, about 
dismantling the college and rebuilding it according to every nev^ theory that^s 
advanced. WeVe talking about iraking small, common-sense changes that improve 
the operation day by day. 

It is important to examine all the fecets of the operation, especially those where 
you might assimie that problems are not likely to exist. Registration might be such 
an area. Bank tellers are trained to be courteous for one simple reason — they want 
their customers to come back. It was reported that 31 percent of the students who 
left a particular community college in Kentucky did so because they were frustrated 
with the registration process. You can now register by telephone at several Florida 
universities. It is always possible to make improvements, and we are more likely to 
look for new approaches if we regard students as customers we would like to have 
return. 

The basic quality assurance tools I discussed earlier can help you get started. The 
line chart lets you know whether it's getting better or worse, the Pareto chart helps 
you. identify the major cause of Oie problem, and the fishbone diagram enables you 
to move in a structured way toward solutions and improvement. The gradually 
emergingphilosophy in education centei^ on improvement of quality and accepting 
the fact that colleges, like ind'^tries, are in a competitive business. The new 
philosophy sees the fundamental importariou of aligning purpose with practice. 
You take the statement of purpose or mission seriouf^ly, and if you aren^t living up 
to it, you change the statement so your professed purpose and actual practices are 
congruent. 
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Inspection vs. Process Improvement 



Deming says the most common response to declining quality is to increase the 
number of inspectors* The most common response in education is to increase the 
nimniber of tests given; thirty-eight states now require some type of assessment 

There is little evidence to show that an increase in testing alone causes any 
improvement Returningtothemanu£acturingexample,£uppose you are product 
widgits and a large number of them are being returned because of def ects* Your 
response is to hire additional quality control people who are stationed at the end of 
the line to weed out the defective widgits. Since this is not very exciting work, they 
become inconsistent in their review after not too long and begin missing all but the 
most obvious defects* If you rely on inspectors to improve quality, you have not 
done mucli to systematize or gain control of the quality control process. And, worst 
of all, you have not addressed the basic problem— the flaw in the production process 
thaf s causing the defects* 

In thinking about institutional quality improvement, it is useful to think of a 
college's organizational structure as a set of interlocking processes with a customer 
at the end of each process* In the admissions process, for example, one major set of 
customers is the faculty who end up with the students* In trying to improve each 
process of the mstitution, we should strive for consistency of output, minimal 
variation in the output of the process* 

QuaHtydoes not come fromirispectionbutfromimprovementof process* All the 
testing instruments in the world from ETS or ACT will do no good imless their 
results arp channeled back to the person teaching the course, unless they provide 
meaningfi 1 information about the effectiveness of the instructional process* Until 
English composition teachers can see exactly the corisistent deficiencies of their 
students, they will not be able to make somid modificatioris in curriculum* 

Quality improvement is a slow, continual, step-by-step process* Positive change 
will not occur ovenught, in sbc months^ perhaps not even in sbc years* Some say it 
takes a decade to imbed a quality improvement process into the structure and fiber 
of a place* You have to be willing to work with all kinds of people to make things 
better a little bit at a time* If you expect rapid tumaroimd and quick fixes, you wall 
be constantly frustrated* 

I see testing as being at a crossroads* Though the assessment movement is bigger 
now th^ it ever has been, I also sense a certain stagnation* Unless the testing 
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movement can be translated into institutional action and reform, it may disappear 
from the scene like a lot of other temporary movements or feds. The feult does not 
lie in the tests themselves: both ETS and ACT are producing better and better 
instruments, and we need them to help us imderstand whaf s taking place in 
education. But we need to become much better at arraying the information and 
using it to make soimd management decisions. 

One facet of soimd management is building solid working relationships. Ford 
Motor Company is going to change its traditional reliance on multiple suppliers of 
steel. The argument for multiple suppliers is that the competition among them will 
enable you to always get your product at the lowest price. But the dovm side is that 
if a supplier knows they may be imderbid and lose a contract in any given year, they 
willbe very reluctant to sink any money into improving their product or process. So 
Ford has decided to try the single-supplier approach in hopes of building a better 
relationship, which they feel will also lead to a hig^er-quaUty product line. 

Likewise, in education, we should know who our primary suppliers are and do 
our utmost to build strong personal relationships witfi them. Do you have a solid 
relationship with the primary high schools and agencies in your area? If a high 
school has sent you 80 students in the last two years, have you shared with that 
school information about the performance of their graduates? Have you had 
guidance coxmselors from that school visit your institution? 

The point is that once you've identified your primary suppliers and come to see 
your school as part of many interrelated processes, you've begun to collect data. You 
have begun a commitment to improving product and service. 

Continuity, Training, and Leadership 

If it's true that we learn most from our mistakes, then I've learned a lot. One of 
the biggest mistakes I used to make was to go to school and try to convince them to 
adopt an entirely new curriculum based on the latest method I'd been involved with. 
It never worked. You must accept whaf s in place and work to improve it 
incrementally. Persistence and common sense are much more important than 
methodology. 

The key is creating and building the spirit to improve. Once you have that, then 
you can move to reworking the constituent parts. Rather than starting with the goals 
at the president's level, you are better off to look at the specific goals of the English 
department, the admissions department, the placement office, and the counseling 
center. 
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Training and expertise are difficult to establish, and sometimes even harder to 
maintain. I learned this lesson painfully at Florida State— I call it the "Arizona State 
Syndrome." We had a person who had introduced a new auto-tutorial teaching 
approach in geology. He quickly establishedanationalreputationfor his innovative 
tedmiques and was snatched away for more money by Arizona State. Since we had 
no one trained in the new methods, we reverted to the old instructional methods. In 
essence, we had spent at least $45,000 and secured no lasting gain for our investment. 

Once you have found an effective approach or program, it is critical that you 
implement formal, ongoing training so the expertise will not be lost. I stress formal 
training because if new teachers come on the job and pick things up from the staff 
in place, there is no guaranfee they will adopt the best practices; they may simply be 
influenced by faculty members who have been on staff the longest. Formal training 
is part and parcel of ensuringthatthecurriculimi and instructional practices of your 
institution align with your statement of purpose. 

According to Deming, if a work force behaves in a relatively consistent way over 
a long period of time, management is 85 percent responsible for that behavior; no 
amoimt of preaching or pressure to change will improve things more than 15 
percent. Deming contends that the main problem with American manufacturing 
and service industries is not lazy workers— if s poor leadership and management. 
This comes from a man who's now ninety and has spent a large part of his life 
stud3dng and comparing the world's management systems. In essence he is saying 
that whatever outcomes you find in a manufacturing process, service industry, or 
whatever type of organization, there is a management philosophy and system that 
is responsible for those outcomes; to change the outcomes you have to change the 
system, from the top down. 

Whether formally articulated or not, each institution has a teaching "system." I 
think the typical scenario is this: Mew teachers come in the first year with lots of new 
ideas to try, but sooner or later get worn down— if they meet resistance— and settle 
into doing it according to the established system. Here's a specific example to 
illustrate this phenomenon. 

At one small school where I worked, I was interested in us5ng a lot of audiovisual 
equipment. To get a VCR and a TV monitor to my classroom, I had to arrange for 
it every day. Then someone had to load the equipment onto a cart, roll it down the 
hallway of the administration building, enlist the help of another student to help lift 
it up five conaete steps, roll it over an asphalt driveway, take it up the steps into the 
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building where I was teaching, then take it down a long flight of steps into the 
amphitheater where I held my class— and set it up. I noticed that most of my 
colleagues weren't using audiovisual aids. I wonder why. This shows how the 
procedures of an established system can work against the best intentions of the 
instructional staff. 

If you expect change, you must have leadership. Even though there are many 
fine leaders in American higher education, we must become more conscious of the 
importance of leadership. There are many great leaders in the commimity college 
sector,butrmnotfamiliarenoughwiththetwo-yearscene to give specific examples 
without offendingsomeone. I willmention one example from the four-year sector — 
Charles McClain at Northeast Missouri State. 

Northeast Missouri has been written about a great deal for its innovative uses of 
assessment, but what I feel makes the institution outstanding is the leadership style 
of McClain and his associates. He really cares about his students and faculty. If 
you're a speech teacher and one of y o'or students does well at a contest, you'll get a 
letter congratulating you on the p^f onnance of your student. 

McClain keeps informed of institutional specifics and finds ways to fix specific 
problems. One year, for example, a survey of PE students revealed that the advising 
services were viewed very negatively. The PE department proposed specific 
solutions and received funds to implement them. Three years later the rating of 
advising by PE students was way up. 

McClain believes in cooperation and balance between athletics and academics. 
When another assessment showed that Northeast Missouri PE students were below 
par in vmting skills, the head of the PE department worked with the head of the 
English department to create a tailored writing skills program. If the football team 
is doing well on the field, McClain checks on their performance in the classroom. If 
the team is losing but performing well academically, the coach knows his job is 
secure. 

Leadership determines how assessment results are used. Many faculty members 
are wary of assessment because they know that numbers are more often used to get 
them rather than help them. A common— and often justified— perception is that 
when administrators start talking nimibers, that talk will lead to the elimination of 
staff. 
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But numbers can and should be used to improve quality, not just determine the 
least expensive way to operate. Deming believes that 97 percent of people want to 
play it straight and do it right; only 3 percent are inherently ia2y. 

Deming's Point Eight is: Drive out fear. This is a very real factor in quality 
improvement efforts. One fear shared by many is that the whole undertaking is 
hopeless from the start. Others fear that if they reveal their ideas others will steal 
them and use them for their own advancement. 

Things cannot improve tmtil you drive out fear. Fear inhibits commimication in 
many ways: people who are afraid are reluctant to volunteer ideas or admit 
mistakes; people (or departments) who are afraid tend to stick to themselves and 
resist appeals for cooperating and working together. 

Thebairier of fearmust be broken down before different fiactions can trust each 
other and work for their common good. The classic example in colleges is of feculty 
and admissions people. The faculty is certain that quality would go up if only 
adnxissions would send them better students; admissions counters that feculty 
wotild go hungry if only Harvard-level students were admitted. What neither 
wants to accept is that, in most schools, neither the types of students admitted nor 
the teaching circumstances are likely to change drastically. If the twofections could 
communicate and cooperate, they could jointly begin to improve existing realities. 
But that is very difficult to achieve, and the impetus must come from the top-level 
leadership. 

The largest impediment to establishing quality improvement programs in 
industry is middle management; middle managers are often resistant to new ideas 
and procedures. I'm sure you can think of similar pockets of resistance in your 
schools— department heads, perhaps, or registrars who are tied to established 
procedures and resent the notion of cooperation or of someone from "the outside" 
telling them what to do. Again, the trick is to introduce new concepts or strategies 
in a way that is not threatening. 

Education has always been fond of fads and catch phrases. Five years ago 
everyone was talking about "excellence in education." Now we're caught up in 
"quality improvement" and "outcomes assessment." Over the years you've heard 
such wisdom as "don't work harder, work smarter," "do it right the first time," 
"don'tfbcitifitain't broke," and soon. However, paying lip service tosuchplu-ases 
is not the same as changing things for the better. 
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Numbers and Results 



Many5H±ookmeasurethesuccessofprograins-K)rde^ 
on the basis of quotas. Or if a course does not attract a certain minimum number of 
students, it is cancelled. While use of quotas is necessary to some extent, excessive 
relianceonquotascanalsoleadtonegativeresults. Theremaybeatendencytomake 
tests too easy or to falsify results just to meet numerical requiiements. 

Deming thinks that imposing numerical expectations on people is 
counterproductive; they could produce the same result— if not better results— 
without the artificial quotas. He says the main reason American managers manage 
by numbers is that they don'timderstand what they're supposed to do, but theycan 
count. One of the most serious problems in American industry is taking college 
graduates and giving them responsibility for a task they have never performed. 
Likewise, in education we appoint administrators in areas where they have never 
taught. To conceal their ignorance, they fell back on quotas and say: "Give me so 
many of this, so many graduates in this program. I don't care how you do it, but give 
me X nimiber of them." Even worse, we expect yoimg experts in English or biology 
to teach well without any relevant or practical training in teaching. 

It is the leader's job to care about how, to provide the necessary control and 
direction. Here's anoperationaldefinitionof 'leader": onewho arranges circumstances 
tohelp people andmachines do their jobsbetter. If sthatsimple— andthatdifficuit. Your 
president should be arranging your drcumstances so you can do your job more 
efifectively. You should be doing the same for the people you work with. And, 
believe me, setting niunerical goals is not the key. 

The key may be instilling, encouraging, or removing barriers to pride of 
workmanship. I recall a Deming story about a Ford plant, where a couple of 
engineers found a solution to a major problem in the Escort automatic transmission— 
a solutionthatsaved Ford close to a billion dollars. When asked if they had persisted 
in finding solution because of possible rewards or incentives, the engineers replied, 
"No, we did it because it was the right thing to do." Again, going back to one of 
Deming' s points, never tmderestimate the inherent drive in people to do the right 
thing. 

It is worth emphasizing this key conclusion of Deming: most people want to do a 
good job. This is not a theory or a wish of Deming's; it is a conclusion he has reached 
after studying a great variety of industries for several decades and consulting with 
some of the biggest industrial leaders in the world. What keeps people from doing 



32 



a good job in their work environment? Barriers to taking pride in their work. Unless 
people are given the encouragement and opportunity to do things the right way, 
they will eventually give in and do things according to the system. 

In conclusion, it is the role of the institutional leaders to identify the factors in the 
working environment that keep people from taking pride in their work and to 
i^ovethosebaniers.Increasin^y,institutionalleadersarereali2angthat^ 
if employed thoughtfully, can play an important part in this activity. What I've tried 
to stress is that obtaining the results is only half the battle. Nothing of substance will 
happen (1) until the top leadership of your institution sees assessment not as an end 
in itself but as the first step in the quality improvement process, (2) until the 
leadership becomes committed to using systematic and contibnuous assessment as 
the way to find out how well the parts of the educational machine are working, (3) 
imtil the leadership understands the potential of assessment to serve as a powerful 
tool for more effective management. 
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DAYl 



GROUP DISCUSSION AND REPORTS 

Groups were asked to discuss institutional effectiveness and to arrive at a 
definition of it. The suggestion was that each individual has a personal definition 
of institutional effectiveness that probably differs— at least somewhat— from that of 
his or her institution; similarly, each institution's definition probably has unique 
elements. Thetaskwastoarriveatagroupdefinition. Since that was likely to require 
some negotiating, groups were asked to report on key differences and to reflect on 
how they arrived at their group definition. 

At least one concept or "definition" of institutional effectiveness appeared in 
each group's report: the accomplishment of goals or fulfiUment of purpose and 
mission. Group discussions tended to focus on the question of whose purposes or 
goals must be accomplished or given the highest priority. 

"Suburban" Institutions 

The first report back to the conference was by Roger Van Winkle, President of 
Massachusetts Bay Community College and a member of the "suburban" college 
group. The members of Roger's group decided that institutional effectiveness could 
be defined from the perspective of either institutions or students, but the definitions 
would share major elements. The following ideas represent this group's consensus 
of what institutional effectiveness means from the institutional perspective: 

1 . Students accomplish the goals they set for themselves or, with the help of the 
institution, they revise their goals and accomplish the new goals. 

2. The institution must provide the environment and appropriate support for 
both students and faculty to be successful and effective. In lieu of that, the 
institution must facilitate the faculty in remaining up-to-date or current in 
their respective disciplines. 

3. The institution must be doing what it says it is doing. 

4. The cimiculum of the institution must be relevant to 'iie commtmity served. 

5. There is a certain vigor or esprit de corps at the institution. 

Students, in the view of this group, would define institutional effectiveness the 
same as the c»llege. This group felt, however, that the following aspects of 
institutional effectiveness would be especially important to a student: 
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1. Job preparedness 

2. Transfer preparation and success 

3. Does the college provide help when it is needed? Is positive personal 
treatment the norm? 

4. Convenience oi attendance and of the standard operating procedures of the 

coUege.Forexample,howweUdoestheschedulefitwiththestudent'sfamily 
and job? 

5. Can the institution meet both social and academic needs for the student? 

6. Whatisthe "comfortlevel" forstudeuts— particularly for nunority students, 
lught-class students, part-time worldng students, and so forth? What type of 
"bonding" takes place so that the student can develop a sense of identity with 
the institution? 

Some of the general topics discussed in arriving at these key points are as 
follows: 

1. How do students set and meet goals? 

2. The institution may encoxmter conflicts and contradictions in the process of 
attempting to provide an environment and atmosphere conducive to student 
success. For example, registration by telephone is easier and more convenient 
for the student, but it entails a loss of personal contact. 

3. It is difficult for the institution to gauge its effectiveness through student 
behavior. For example, earning credits may be more important to the student 
than earning an actual degree because many transfer earlier or come for some 
reason other than a degree. You cannot determine whether the institution is 
effective imless you can determine what the actual goals of the students are, 
and that may be difficult. The idea of asking for student goals on the 
application form seems good, but students do not always seem to xmderstand 
what kind of information is needed or wanted when filling out applications 
and registration forms. 

4. Do we train for now or for the future? Skills change and needs change. How 
do we teach coping skills so that our students can be prepared for the future? 

Small Institutions 

Nancy Coleman, the Director of Assessment at Orangeburg-Calhoxm Technical 
College served as the reporter for this group. Some of the colleges in this group are 
already extensively involved in using student outcomes measures to assess 
institutional effectiveness. This group had two definitions of institutional 
effectiveness to share: 
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1. How well you do what you say you are going to do. 
(This is the John Roueche definition.) 

2. Comparing institutional performance to institutional goals. 

In discussing definition #1, the participants realized chat what you say you are 
going to do and how well you do it can be tied to the size of the institution and the 
level of its resources. Fixed costs make up a larger percentage of the total costs of 
smaller institutions, and coming up with money to implement an assessment of 
institutional effectiveness may be difficult. In addition, larger institutions have 
larger pools of human resources and generally more expertise to draw upon. 

Definition #2 brings up the importance of a mis.sion statement. A good mission 
statement is focused and explicitly describes what you want to do— what you want 
to measure. The mission statement must guide the assessment process. 

Institutional effectiveness must be measured over a period of time. A feedback 
of information gathered in this process is essential to improvement for the institution. 
There must be broad participation in the process. 

To measure institutional effectiveness, an assessment process is needed that 
spreads throughout the institution. Institutional effectiveness must be viewed first, 
as a whole; then the targets (subsets of the whole) of that effectiveness can be 
explored. Targets include (1) student success, (2) the community effect, (3) fiscal 
policies, and (4) the demographic profile. 

Discussion of definition #2 led to the conclusion that comparing the performance 
tothegoakorpurposesoftheinstitutioncaimotreaUybedoneiinlessthed^ 
profile of the service area is taken into accotmt — ^where students are, what kind of 
students are there, where they go, and so forth. 

It is important to remember that all students have potential for success. Student 
success is defined in terms of salary and the impact on future salary levels, job 
opportunities, and a sense of having been made to be a better (more capable) person 
on completion. We can add to student success if we know why students come (their 
goals). We must let them make plans to achieve their goals, but we can be proactive 
as well. We must get quantitative information tinrough assessments, and we must 
share this information wildely. 

GroupdiscussionendedonthequestionofhowweUtheinstitutioncommiinicates 
its goals, purposes, or mission to the community. Whether definition^ ordefinition 
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#2 is used, it is important to comr unicate to the cominuiuty what you plan to do — 
your goals, mission, or purposes. 

Mid-sized Institutions 

Sandi Oliver, Director of Admissions at Midlands Technical College, seived as 
recorder for this group, and Tom Gill of Oiemeketa Commimity College presented 
the report to the conference. This group was impressed by the concept of "aligning" 
activities and purposes that John Harris suggested. Their first point in defining 
institutional effectiveness is the alignment of activities with purposes. But that 
presupposes a clear piupose and the ability to define that purpose dearly. When 
reporting to the conference, Tom stressed first that their definition is perhaps that 
institutional effectiveness is doing what the institution intends to do. Another point 
that he included from their discussion is "progressive realization of the need to 
improve." Thatprogressionis ''fueledby ablend of internal and extemaiforces such 
as external mandates, public images of the mission of the institution, values of the 
institutiorvstudentgoalsandpotentials,feculty,andresource^ "Effective institutions 
do something about their realizations and adjust" 

A review of the points recorded by Sandi provides insight mto how the group 
discussion evolved and how they moved from the aligiunent concept to a more 
complex definition. 

1. Align activities with puGcpose. (Presupposes a clear idea.) 

2. Must be able to define the purposes. 

3. Align activities, per se. 

4. When students can, with their abilities and educational goals, use the 
resources of the institution to achieve the desired goals (and achieve their 
potential). 

5. When students use a sendee and get back from that service what is desired 
or required. (This is a corisumer-oriented defiiution. Js there a need for a 
college to provide more than a consumer function to meet the student's 
definition of effectiveness?) 

6. When the faculty get what they desire/require from the system. 

7. The link to the institutional mission must be specified. This requires the 
following: 

a. Identify the mission. 

b. Identify the components ol the mission and the goals of the institution that 
contribute to the mission. 
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c. Assess and evaluate the extent to which the mission is achieved. 

d. Use the results to make improvements; there must be a continuous 
process of planning and evaluation. 

8. When the institution expands the options from which students can develop 
and achieve their potential, 

9. When there is a commonly shared sense of belonging. 

10. It is necessary to use both quantitative and qualitative measures to assess 
effectiveness. 

11. The fectors that contribute to institutional effectiveness must be defined to 
determine how effective the college has been at achieving its mission. The 
critical success factors must be identified/ and/or the values that contribute 
to the mission must be identified and defined (operationalized) so that it is 
possible to assess the extent to which these fectors are being achieved. 

12. Institutional effectiveness is a blend of critical success factors and values; it 
is necessary to integrate a variety of variables and measures to determine if 
effectiveness is the goal. 

13. What does it mean to ''achieve the institutional mission''? Do we think of 
achievement of mission as continuous improvement in general or do we think 
of it in more specific terms such as the identification of a targeted goal and 
determining through measurement that it has been met? 

14. There is an important distinction between continual improvement and 
improvement of a spedfic amount in a specific time. 

15. Institutional effectiveness implies the progressive reaIi2;ation of the success 
of predetermined goals. 

15. A measure or assessment of institutional effectiveness involves a blend 
between internal goals and external impositions. 

Urban Multi-campus Institutions 

Last to report back to the conference on their discussion of a definition of 
institutional effectiveness were the representatives of larger, urban ir.i3iitutions with 
more than one campus. The reporter for this group was Betty Pollard from St. Louis 
Community Colleges. This group also drew from some of the concepts presented 
by John Harris. Again, the notes from the group's discussion indicate that in trying 
to arrive at a definition or some necessary components of a definition of institutional 
effectiveness, the group had to struggle with defining more terms and concepts. 

The key to defining institutional effectiveness is to define it in terms of the 
mission and purpose of the institution. This involves the following: 
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1. Mating purpose with performance. 

2. Mgning purpose with performance. 

3. UtiUzingresoiircesinthebestwaytopromoteeffidencyasweUa^ 

4. Meeting commimity expectations. The "80-20 rule'' must apply, ie. the 
purpose must be met 80 percent of the time. 

5. Having a way for institutions to impact students cognitively and affectively. 

In defining institutional effectiveness in relation to institutional purpose, we 
must address the following: 

a. Who shapes the purpose? 

b. Whose purpose is it? The institution's? The student's? That of business and 
industry? 

c. There are local as well as national interests to be served and met. 
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DAY 2 



OUTCOMES ASSESSMENT IN ACTION: 
THii NORTHEAST MISSOURI STATE UNIVERSITY EXPERIENCE 



TERRY B. SMITH 



Dean of the College, Northeast Missouri State University 



Introduction 



Why do we assess students? Assessment involves both students and faailty in 
learning outcomes, raises proper questions about the curriculum, and promotes 
reflection on teaching methods. It enhances academic achievement, stimulates 
student satisfaction, provides anenvironment for long range planning, and facilitates 
program review. Most importantly, and the reason why assessment should be 
entered into in the first place, assessment measurably in^jreases learning. 



There are six steps to building an assessment program. First, design a theoretical 
constructor a model. It can have the appearance of a flow chart, which conceptualizes 
data items, identifies level of analysis, focuses on the purposes of the assessment 
model and links assessment activities with desiied outcomes. 

Second, utilize all data already available. On most campuses archival data on 
demography, student ability levels, student achievement levels, and student surveys 
exist, probably scattered among several offices. Data can be analyzed on a school- 
wide basis, on a discipline basis, on a major basis, and even on the individual student 
basis for advisement ptuposes. 

Third, collect the desired data elemerts and assemble them in a manner easily 
communicated. Fourth, analyze the data by organizational imits of the institution. 
They can be reviewed on a macro (school-wide) level all the way to the micro 
(individual student), und the schools and departments in between. 

Fifth, pro . . access to the data to all levels of the school. The data are better 
presented raw with suggestions on how to analyze it. Pre-analyzed data are often 
seen as "cocked." Sixth, provide longitudinal data for comparison of each imit 
against itself. Several years of data beccne astonishingly revealing. 



Creating an Assessment Program 
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Finally, focus the attention on learning. Data collected support the learning 

process,raiseinstructional andlearningquestions,identify areas to bestrengthened, 
and reward achievement and achievers. Important learning outcomes include: 
general education objectives, satisfaction with instruction, levels of scholastic 
achievement, and academic persistence. Non-academic objectives include: 
placement, leadership, professional, further education, personal, and employer 
satisfaction. 

The likelihood of successfully reaching goals is increased by, first, translatii ig the 
goals into measurable outcomes, then selecting means of assessment that reflect the 
institution's distinctive mission, goals, and objectives. Next, identify the points of 
assessment that best reflect the achievement of goals, then use multiple measures to 
strengthen the vaUdity of assessment (triangulation). Finally, collect data and 
analyze these data with the goal of improving the teaching and learning of students, 
then plan strategies for improvement in various sectors. 

Value Added Assessment at Two-Year Institutions 

Value-Added assessment is referred to by Alexander Astin as 'Talent 
Development." He asserts that value-added outcomes assessment is an ideal 

studentand academic developmenttoolfor institutions with studentbodiesof great 
diversity, including large groups of students ill-prepared for post-secondary work. 
Northeast has affirmed Astin's assertion, finding that, while everyone shows 
"value-added" on the pre/post ACT testing between the freshman and sophomore 
year, the students in the lowest quartile of entering ACT show the most growth in 
ACT score. ^ 

The real value, however, of value-added outcomes assessment rests with the use 
of data for specific program improvement. Northeast Missouri State began its 
Value-Added program in 1973 with voluntary sophomore testing. The program 
was not well enough developed until the late seventies to begin utilizing the data for 
specific program development and improvement. In 1979 the imiversity had 
sufficient data on returning sophomores who had also taken the ACT as then- 
freshman entrance exam to make meaningful comparisons on the four subject areas 
tested by the ACT. Northeast found, most interestingly, that during two of the first 
three years of the 1979-82 period that there was an actual decline in the math section 
of the sophomore ACT when compared with the fireshman test for the same 
students. During the same period student surveys showed a low rating of the 
student ability to understand mathematics. In addition, quantitative scores on 
senior tests were weak. 
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When faculty reviewed these findings they focused on learning inputs and 
outcomes, created several hypotheses about the score declines, tested them, and 
directed their efforts toward the 'Value-subtracted'' problem. Faculty began 
discussing requiring higher mathematic skills across campus. Further, they began 
examining the quality of instruction and grading standards in math classes. 

A fundamental cxirriculimi revision occurred. The mathematics requirement of 
general education was increased from a contemporary math course ("math 
appreciation") to college algebra. Calculus was required for in business 
administration and accoimting majors. A no-credit math lab was introduced that 
students had to pass before taking college algebra. 

Students were advised into the stronger math sequence and the changes were 
monitored and the results were shared generally across campus. In 1982 ACT 
sophomore scores began to increase over freshman scores and continued to increase 
to the present day. Further, there was a rise in the nimiber of smveyed students 
reporting better understanding of mathematics and there were fewer problems in 
upper division classrooms because the math foundations were stronger. 

The English part of the ACT has continuously shown improvement at a higher 
level than any of the other components. However, while solid gains were routinely 
produced ^he imiversity still pursued a revision of the writing philosophy. The 
university now practices writing across the curriculum. More faculty across campus 
are requiring writing. Correct writing became the norm and writing became used 
as a vehicle to enhance thinking. 

English composition I and II changed in content emphasis to include expository, 
research, and organizational writing skills. The writing lab became an extension of 
the compositional thrust. The staff was upgraded to full-time professional staff. A 
proficiency exam was required that was jury-graded. Writing competency was 
made a graduation requirement and writing became a total imiversity experience. 

Student advisement began stressing the importance of writing skills, permitted 
feedback and input, and increased individual student awareness of the importance 
of writing. Student surveys showed increases in student satisfaction with the key 
liberal learning skill of writing effectively. 

Similar data were collected using the ACT COMP. In the sbc competency areas 
the university realized significant gains. Crossreferendng the ACT results with the 
ACT COMP was another example of triangula tion. Not only in subject areas but also 
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in liberal learning skill competencies students were reali2dng measurable benefits 
because of their freshman/ sophomore general education experience. 

Individual students can also be assessed and their growth can be charted with 
considerable accuracy. One student who recently graduated from the university 
showed a more than 30 percentile increase on his sophomore ACT. His percentile 
increase in English was more than 60 percentile. When asked about this increase he 
said that he had done the best he could when taking the ACT in high school and that 
the English composition class that he took was one of his most valuable experiences. 
He attributed the increased score to college instruction. 

Another student who came in with a low ACT showed enormous percentile 
growth— more than 60 percentile on her composite score between freshman and 
sophomore ACT. However, because of poor advisement she was placed in classes 
too advanced for her entering ability level and she flunked out of school. When 
contacted about this, after discovery of the error, she said that she had learned a great 
deal but did not have adequate study skills for success in the classroom. Both her 
sophomore ACT and gpa confirmed this. She was readmitted, graduated two years 
ago, and is now a medical technician. 

Stages of the Assessment Experience 

The Readiness Stage 

When an institution prepares forassessmentits readiness canbe either internally 
or externally motivated. If internally motivated, it has been stimulated by the 
school'sinterestin improvement, indemonstratingeffectiveness,and/orinknowing 
more about its students. Assessment may also be desired as a result of the national 
mood. If readiness is externally motivated it is because of a mandate by a state 
legislature or governing board, or by the local board, or by constituent pressure. 

Benefits are few during the readiness stage and consist largely of the discussion 
ofshared goals and objectives. Costs,ontheotherhand,aremanyai\dhigh. Motives 
are questioned. There is insecurity, tmcertainty, disagreement, frustration, 
misunderstanding, and considerable anxiety. 

Tunecosts are also high. Thediscussionstaketime. The planning and organizing 
are complex and involved. Developing the model is a sophisticated and tedious 
process, as is collecting available data, projecting operational costs, and identifying 
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funding sources. Building legitimaqr is perhaps the highest and most demanding 
cost at this time. 



The Implementation Stage 

When the institution gets its first bit of information the data may be internally 
upsetting. Data are often rejected because the tests are not considered appropriate, 
the surveys are considered unreliable, or the students are not believed to be taking 
either the tests or the survey seriously. Institutional motives are questioned. The 
process of assessment is questioned. Skepticism and denial of evidence occur. If 
feced,theinformationmicovered raises hard questions. The chaUenge is to stay with 
the process. 

The cost/benefit ratio becomes more balanced during this stage. Benefits 
include raising important questions, focusing on effectiveness, stimulating the 
educational agenda, explidtiy stating goals and purposes, and proceeding on an 
innovative comse. 

Considerable arousing of professional interest takes place at thic time. Feedback 
is made available to faculty stimulating discussion on the curriculum. Students can 
begin to use available feedback to recognize the strengths and weaknesses and 
feculty can begin expecting increasing expectations for students. 

Significant costs remain. Many cf the costs in stage one are still present. Some 
new costs include difficulty in interpreting data, recognition of hard truths, 
illmnination of comfortable myths, and lack of pei:sistence and patience. T5me costs 
include time taken to discuss data, to collect it, to build formats \ot ils release, 
communicating successes, improving weaknesses, administering the instruments, 
and, and always, building legitimac;;\ ITiere are obviously monetary costs. The test 
and siurvey instruments are not free. Administration and overhead are real 
expenses, as are computer time and consultant costo. 

The Acceptance Stage 

The faculty, administration and students accept the data. Data have been 
collected for several years now. Multiple measures (triangulation) corroborate 
findings. The data basebeccmes a means for faculty to institute curricular refinements 
andrevisions. Thedataonacademicperformancecrystalizeexpectations. Questions 
on improving student performance direct goals and activities of other campus 
oiBfices. The data help the campus conmumity focus on student learning. Discussion 
on goals and achievement increases involvement. 
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Improvements begin showing up as a result of changes. The longitudinal data 
reveal trends related to institutional mission. Benefits begin to outweigh costs. 
Assessment continues to raise questions but now also suggest answers. Assessment 
is integrated into all phases of institutional life, conveying a positive spirit to 
improve. All staff and feculty are motivated toward a common goal. 

Assessment provides direction in planning, clarifies the mission, strengthens 
effectiveness, provides internal and external recognition, and rewards improvement 
in achievement. It promotes discussion on schoiaiship, influences hiring and 
allocation of resources dedsions, and produces cost effectiveness. 

The assessment process increases discussion on teaeliing and learning and 
promotes action on teaching and learning. It fosters clear thinking about the 
curriculum, emphasizes the learning process, and involves faculty in analysis of 
data. It provides direction for solutions, defines the program more clearly, and 
facilitates program review. 

For students,assessmentincreasesexpectations,increases involvement, promotes 
student discussion on goals, raises confidence and achievement, and directs 
hnprovemant toward the total student. 

Costs remain, however. There is TiO quick success. Lingering concerns remain 
about possible misuse of data. Time is still taken up improving collegial cooperation, 
investigating the data, planning improvement strategies, making data available and 
commtmicating. Financial costs remain as well. 

Hie Commitment Stage 

The fourth and final stage is commitment. Costs remain. There is still time 
involved in planning and reassessing and in initiating innovative ideas to enhance 
scholarship. There are still monetary costs: instruments, administration, and 
overhead. 

But in the commitment stage benefits overwhelmingly outweigh costs. Change 
becomeseasier,coUegiahtyoncampusisenhanced and there ismcreasedcom^ 
on every level toward academic achievement. Satisfaction toward the school is 
improved. Amorescholarlyenvironmentiscreated. Fuuidingbecomes performance- 
driven. 
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Built into academic programs is continuous review and revision. Faculty 
development increases, student-faculty interaction increases, cooperative learning 
improves, time on task increases, and data provided to students help direct goals. 

Eight Ingredients for Success 

There are eight ingredients for a successful assessment program. First, involve 
as many students in the institution as possible, hopefully all. Universality obviates 
sampling and the ''guinea pig" problem and promotes more student ownership. 
Second, always use multiple measures. Triangulation guarantees a more 
sophisticated and holistic view and ameliorates concerns about oversimplification. 
Combining standardized tests with attitude surveys and traditional assessment 
devices (student course evaluations, peer reviews, anecdotes, grade point average 
studies, retention studies, etc,) creates u good mix. 

Third, cultivate feculty ownership of assessment. Assessment will not v/ork 
r^ess faculty support its aims and are sure they are not threatened by it. Faculty 
attitudes si^iificantly affect student attitudes about assessment. Fourth, sell 
assessmait to students both at the recruitment and enrolled stages. Students who 
come to the institution knowledgeable about its assessment program and goals will 
be more cooperative. 

Further, it is vital to provide optimal conditions for assessment activities and to 
attend to student concerns about the process in order to maximize student 
performance on assessment instruments. 

Fifth, assess students, programs and institution. In other words, assess for all 
appropriate levels and never assess inappropriately; i.e., never collect data that can 
isolate individual faculty, even though it may be technically possible to do so. 

Sixth, make data and analyses imderstandable. Key constituencies will be 
suspicious of esoteric or technical reports of assessment activities. An advantage of 
'Value Added" assessment is the amenability of its data to basic "plus/minus/ 
zero" array. Either an individual student "adds value," or she doesn't. Either 
groups of students show better math skills after hvo years in school, or they don't. 
Either students feel more satisfied about faculty advising this year than they did last, 
or they don't. 

Seventh, be flexible. If one instrument no longer serves your assessment needs, 
scrap it. Be vigilant for new instruments. The testing companies are getting serious 
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about Value-Added instruments (EIS'sAcadenu^ 

Encourage faculty experimentation. Finally, use data to improve, not to punish. A 
quick way to kill an assessment program is to use results negatively for promotion, 
tenure, and budget decisions. 

Conclusion 

Assessment is no longer experimental or avant-garde. It just IS and it has made 
a measurable difference at institutions like Northeast Missouri State that have been 
doing outcomes assessment for awhile. 

The common denominators of these schools are that they wanted to know what 
was REALLY happening in their classrooms and labs and libraries and faculty 
offices, persuaded the faculty and students that the quest was important, and studc 
with it long enough that assessment eventually became not an end in itself but 
simply another vital means of achieving quality education. 
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GROUP DISCUSSION AND REPORTS 

Conference partidp ints were again asked to form discussion groups according 
to the type of institution they represented: small, mid-sized, suburban, or urban 
multi-campus. The objective of the task this time was to de&nevalue-addedassessment 
Participants were urged to focus on v;hat value-added assessment means, how to do 
it, what issues are involved, how hindrances or problems with doing value-added 
assessment may be overcome, and how value-added assessment should be used. 

All the groups shared common themes: the difficulty in isolating or differentiating 
the impact of the institution (value-added) from other contributing factors; the costs, 
not just financial, involved in carrying out this type of assessment; logistical 
problems relating to tracking students to have a fbllow-up assessment, and especially 
how these problems will differ from those of four-year institutions; getting students 
to participate and to perform so that the results serve as a valid indicator of change; 
and generating enough interest and commitment on campus to implement this type 
of assessment. 

Urban Multi-campus Institutions 

Representatives of these institutions were immediately concerned with the 
terminology. Barbara Keener, Dean of Academic .Affairs and Community College 
Relations at the University of Florida and representing NCSD as the secretary of the 
coiincil, served as the reporter for this session. She noted that the first concern was 
with the use of "value-added" as a noun rather than as an adjective. 

Tlds group was more comfortable with the concept of "talent development" than 
with "value-added." 'Talent development" seems to avoid the economic overtones 
suggested by "value-added" and does not carry the possible negative connotations 
of "value subtracted." In either case, the emphasis is on what the institution does for 
the student, an emphasis that requires one to examine the balance of other impacts 
on the student's life in order to sort out the specific impact of the institution. 

A concern of the group was with communications, or the lack thereof, between 
administrative and instructional staff with respect to the role of the college. They 
questioned how administrators and faculty can talk to each other, what 'language" 
should be used or concepts should be borrowed to facilitate communication? 
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Another concern regarding the role of the faculty was with how individual 
faculty functions correlate with assessment: What happens in the classroom that 
directly relates to test results? It is important to understand what is being produced 
when conducting an assessment of this type (value-added or talent development). 
One must deal with specifics. 

Some of the benefits of this type of assessment include: 

a. Student preparedness 

b. Documentation that provides self-satisfaction 

c. Marketing the institution's impact 

d. Concrete evidence of an impact 

e. Formative evaluation— it provides the opportunity to talk with the students 
before ^ey finish, to give feedback along the way, and help them improve 
over time. This points to the dififerences between program development and 
student development. 

f . Helps identify patterns and trends 

g. Facilitates greater communication on why you are doing what you are doing 
The following problems or obstacles were identified: 

1 . Costs— both in dollars and time. Also, there are emotional costs, such as fear 
and development of trust, to take into account 

2. Accurate tracking. Acaediting agencies impose mandates; and there are 

other requirements,such asstate funding formulae, thatmustbe considered— 
in addition to developing a tracking procedure appropriate to this type of 
assessment. 

3. Responsibility for activation of implementation— v/'io can do it? 

• A successful value added/talent development assessment program 
requires faculty support, involvement, and commitment. 

• Student participation and commitment are also "musts." 

• The support of the president or chief executive officer is critical. 

• The computer services area mitst be committed. 

• Expertise in the use of the research is needed 
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'^id-sized'' Institutions 



Delinda Cannon, Director of Planning and Administration at Midlands Technical 
College, served as the reporter for this session. The major focus of this group was 
to isolate the problems and challenges for two-year institutions and to discuss which 
components of the value-added concept are unique to the conununity /technical/ 
jimior college setting. The problems and challenges arc outlined here. 

1. Cohorts/change/retention^ Discussion first revolved around the fact that 
the student population for many t. /o-year institutions is rather fluid. Value- 
added assessment is made more difficult because of the need to identify a 
cohort of large enough size to lend research support. 

2. Motivation to test, especially post-test. It is not as easy to get students to test 
when they are finishing their program with you. 

3. Identification of core courses that students must have taken. If growth or 
value-added is to be attributed to the institution, then it must be possible to 
identify courses that serve as the core curriculum. 

4. Open-door admissions, entry-level diversity of skills, backgrounds, and so 
forth. This point relates to point #1 and to the way value added is to be 
measured. 

5. Breadth of conununity college missions. How well does value-added 
assessment contribute to the broad range of missions? 

6. Resources. Who will pay? 

7. Indirect costs. One must take into account the use of staff resources to 
administer tests and follow-up. 

8. Acceptance of the assessment procedure. The planning stages and strategies 
may be more complex, but acceptance may be easier to achieve in a two-year 
college. 

9. Timing of exit assessment. It will be difficult toknowwhen to administer an 
"exit" assessment. This relates to the retention issue already identified and 
also to the core curriculum issue. 

10. Difficulty of measuring "quality of life'' gains in two years. 

11. Confoimding effects of intervening life experience factors. 

12. Will assessment be around in 10 years? Yes, because: 

a. living standards /a^ increasingly based on quality. 

b. global influences force us into competition with an increasingly 
sophisticated global society. 

13. Can we let the marketplace determine our survival? We have political 
realities to deal with; one is that public institutions are not solely supported 
by the local conununity in which they are located. 



"Suburban" Institutions 



The representatives of suburban institutions adopted a strategy remarkably 
similar to that of the midsized schools just reported. This report identifies the 
concerns of the group regarding value-added assessment and strategies that have 
the potential to resolve some of the concerns and issues. This group started 
discussion by questioning how value-added assessment will work on two-year 
college campuses, compared to the experiences of four-year colleges. 

Concerns identified include: 

1* Retention/persistence — having students around at the end to test: How do 
you judge when general education requirements have been fulfilled? 

Z Part-time students: Can you really attribute value-added to their interaction 
with the institution? 

3. Multiple campus enrolhnents: How do you attribute value-added to the 
institution when the student is enrolled at other colleges as well? 

4. Student motivation— getting students to take the test: How do you "require" 
students to be test participants at exit? 

5. Logistics — ^length of the test process, tracking students. 

6. Reverse transfers — ^many students have previous college credits, particularly 
in general education areas. How do you measure value-added by a specific 
institution? 

7. Computer capacity for tracking and storing data/performing data analyses 
necessary for value-added assessment. 

8. Special populations such as returning learners. 

Possible strategies suggested: 

1. Inform ACT consultant for each demonstration site of the institutional 
capacil/ for tracking. The ACT person can serve as a resource. 

2. Studentmotivat'm:requiringassessmentisapossiblesolution. If assessment 
is required, it becomes an expectation on campus. Be sure students know 
why the assessment is being done. 

3. Student persistence/retention — getting students to take the test at the "end." 
Retest by credit hours completed; through public relations efforts, advising 
information, and so forth, it wiQ be possible to "program" the college 
community and student body to expect exit testing. 

4. If institutions have needs for additional instruments (such as special 
questionnaires), they should communicate these needs to ACT consultants. 
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5. We cannot wait for perfect solutions, we have to get started. 

6. As more information becomes available to institutions involved in this type 
of assessment, it will become integrated into the academic environment and 
standard operating procedures. 

During discussion, the suggestion was made to have a student orientation or an 
assessment course. There is a growing national trend to do this- It means that 
assessment can take place, student orientation can take place, study skills can be 
assessed and taught, career development and plaiming can take place, and library/ 
research skills can be learned. 

Sma.U Institutions 

Representatives of the small institutions contributed to the whole group's 
thoughts of value-added assessment by concentrating much of their discussion on 
how the assessment will be used. 

Discussion first centered on defining value-added. The group felt that an 
appropriate definition should treat value-added as a concept and also provide a 
means for operationalizing the term, le., for providing specific calculations to 
measure value-added. Institutions must determine how this information can be 
effectively communicated to feculty, staff, students, governing boards, and the 
community at Ifirge. 

Just as was the case for the other three groups, this group of representatives 
discussed how to motivate students to take the tests and to perform to their 
maximum capacity on the tests. 

The strategies they suggested include: 

1 . Make it a requirement for graduation/program completion. 

2. Get faculty and advise pport. 

3. Take advantage of the ically cooperative nature of students* 

4. Provide a good environment for testing. 

How will measurements of value-added be used in the 1990s? Here are their 
ideas: 

1. Academic evaluations of programs and aivising will emphasize 
improvement, not punishment. 



2. Groups of students with high value-added can be studied and analyzed to 
determine what made the difference for them— what spells success. 

3. The value-added structure will be further expanded and modBed to better 
facilitate value-added assessment of nondegree programs. 

The group discussed the feet that during the 1990s there will have been an 
adequate amount of tiine for the longitudinal studies to ''bear fruit" or show results. 
It takes some time to work out the conceptual and operational meaiung of value- 
added. 

Finally, the group discussed the feet that how value-added assessments are used 
will depend o^i whether the results are positive or negative. Negative results might 
provide the basis for a campaign to get increased funding in order to address the 
problems and shortcomings responsible for this. Positive results, on the othc*r hand, 
can be used asapublicrelations tool. Two-year institutions canreportthese positive 
results back to their feeder high schools and to other high schools from which they 
would like to attract students. Another use of positive results might involve a 
''guarantee'' to adult learners that they will get what they came for— i.e., will get 
their time and mone/s worth. 
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STUDENT ASSESSMENT PRACTICES IN THE COMMUNITY COLLEGE: 
CURRENT STATE, DESIRED STATE, GETTING THERE FROM HERE 

KAYM.MCCLENNEY 
President, MO Educational Consultants 

The concept and the practice of assessment are not new to community colleges; 
on the contrary, the nation's community, jimior, and technical colleges have for 
Eiany years been involved in a variety of assessment activities. Nonetheless, there 
is certainly evident across the country a new rhetoric of assessment, a new urgency 
in discussions and debates, an accelerated pace in the development and 
implementation of assessment methods and programs— all attributable, at least in 
part, to the recent national emphasis on improved quality and accountability in 
higher education. In the following discussion, attention will be given to (1) a brief 
overview of the "current state" of assessment in the community college, (2) some 
propositions regarding desired futiu-e characteristics and directions for community 
college assessment programs, and (3) a few key considerations in the development 
and implementation of assessment programs on local campuses. 

The Current State 

From the recent plethora of reports and studies which have focused attention on 
American higher education, one paragraph stands out as a succinct summary of the 
public and professional sentiment that lies behind the current assessment 
"movement:" 

One of the most remarkable and scandalous aspects of American higher 
education is the absence of traditions, practices, and methods of 
institutional and social accountability. How can colleges and universities 
assure the American people and themselves that they are doing what they 
say they are doing? How does anyone know that the curriculimi really 
"works?" There must be ways of demonstrating to state legislatures, 
students, and the public at large that the colleges know what they are 
doing (or do not know) and that they are doing it well (or poorly) 
(Association of American Colleges, 1985). 

Surveys now indicate that almost half of the nation's public colleges and 
universities areoperatingunderastate-levelassessmentmandate,andthatpercentage 
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is expected to increase substantially each year. The state initiatives come in many 
fonns, including general calls for assessment, required submission of institutional 
assessment plans, and prescriptive mandates linked to funding. The regional 
accrediting associations, led by the Southern Association of Colleges and Schools, 
have added impetus to the movement through their newly strengthened emphasis 
on institutional effectiveness— with special focus on the assessment of educational 
outcomes. 

It will come as a surprise to hardly anyone that there is both "good news" and 
"bad news" associated with all of this assessment activity. The bad news includes 
thereahtythattheextenialniandates— whether fromlegisktures,5oveniingboards, 
state agencies, or other sources— often come with tight deadlines, limited (if any) 
resources for development and implementation of assessment programs, and in 
general, few real extrinsic incentives. Moreover, the assessment task— taken as a 
whole— is awesome; and members of the college community, especially in small 
institutions, may feel woefully inadequate. Compounding the problem is the fact 
that the technology and metikodology of assessment are on the one hand quite 
overwhehning and on the other hand, still primitive. And certainly not least on the 
list of challenges is the level of faculty skepticism— or even downright resistance — 
that often greets campus assessment initiatives. 

So what, then, is the good news? Here at last, cheer the proponents, is a 
movement that has something to do with the fundamental piuposes of community 
colleges— the central tasks of teaching and learning. Too, following the initial years 
ofdiscussionanddebate,thereisaperceivedinevitabiIityattributedtotheassessment 
movement; thus, campus debates are moving quickly away form the question of 
wftef/z^assessmentshouldbe undertaken and focusingmore and more productively 
onthequestionofftowassessmentcanbemostappropriatelyandusefullyimdertaken 
inaparticularcoUegewithaparticular mission, curriculimnL,andstudentpoptiIation. 
hi pursuit of that question, many colleges gratefully discover that the issuers of 
external mandates (e.g., governors, legislators,bureaucrats, etc.) are generally much 
more imderstanding and flexible than is initially assimied— and in feet welcome the 
development of feasible assessment niodels that make sense at the local level. 

There is more good news, including the exciting professional activity- 
conversations, collaborations, consortia— that has developed around assessment 
issues; the demonstrable improvements in curriculimi and teaching that have 
occurred in response to assessment findings; the simple but profound insights 
gained about the relationship between teaching and learning, teacher and learner. 
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Purposes of Assessment 



Provided below is a brief and very general overview of current assessment 
practice in community colleges. Coraments are organized in terms of five major 
purposes of assessment. (See Millman, 1988; Mdntyre, 1988.) Those purposes are: 

1. Placement — ^to admit, advance, and place students 

2. Student learning/development— to advise, coimsel, assist, and teach students 

3. Certification— to confirm or assure student acquisition of certain knowledge 
and/or sldlls 

4. Course and program evaluation — to evaluate and improve courses and 
programs; to plan and budget 

5. Institutional evaluation— to demonstrate overall institutional effectiveness 
and accoimtability 

Placement 

Recent surveys indicate that eight out of ten conunimity colleges are currently 
engaged in assessment of basic skills, primarily f or the purpose of course placement. 
The assessment of basic skills — especially reading, writing, and mathematics — 
represents one of the pxhnarf assessment strengths of community colleges. Not 
surprisingly, given the immense diversity of their student populations, community 
colleges discovered very early the benefits of assessing basic skills upon college 
entry in order both to reduce the range of skill levels in a given classroom and to 
promote student success. 

Despite the substantial experience and relative sophistication of community 
colleges in this area of assessment, there are challengesyet to be resolved. Seriously 
debated on many campuses (and also in the courts) is the question of whether course 
placement, based wholly or partially on assessment restdts, should be advisory or 
mandaf ory. A related concern pertains to the potential impact on minority student 
access to collegiate programs. And on many campuses (including senior colleges 
and universities), systematic placement testing may lead to a massive increase in the 
recognized need for remediation and a commensurate need for teachers, classrooms, 
and other resources for developmental education 

Student Learning and Development 

The Alvemo College model of "assessment as learning" is perhaps the preeminent 
example of a?" ssment conducted primarily to promote, evaluate, and document 




the learning and development of individual students. (See, for example, Loacker, et 
fl/,1986.) Though widelyadmired,themodelisseldomfullyreplicated. Community 
college observers dte in particular the perceived problems of high cost (in terms of 
faculty time) and the immense diversity of commimity college programs, studenu . 
and learning environments. Examples of assessment as learning can be found, 
however, in conunimity college programs with significant clinical or performance 
components (e.g., programs in allied health, performing arts, electronics, etc.). 

Also noteworthy is the current interest in the improvement of classroom 
evaluation, which is perhaps best illustrated by K. Patricia Cross' work in promoting 
the concept of the "classroom researcher." (See, for example. Cross and Angelo, 
1988.) Cross is valiantly attempting to retum the focus of assessment to the 
classroom — ^the primary teaching-learning relationships — ^by equipping feculty 
with research and evaluation skills which willhelp to explicate the linkbetween how 
teachers teach and what students learn. 

Certification and Gatekeeping 

A nxunber of colleges across the country have led the way to describing exit 
competencies in courses and / or programs and then certifying student achievement 
of those competencies. This work, though far from universal, is very promising 
because it gives substantive meaning to grades and credentials. In some fields 
(nursing and alHedhealthoccupations,forexample),Kcensureofcertificationexai^ 
have long served as the gatekeeper for admission to practice. "Rising jtuiior" 
examinations (i.e., competency tests reqtm-ed for admission to upper division study 
and/ or for award of the associate degree) also serve a gatekeeping or certification 
function. Such exams are the subject of significant controversy in states that have 
adopted them, primarily (but not exclusively) because of the potentially 
disadvantaging impact on minority student populations. 

At present, very few conununity colleges have established explicit graduation 
requirements that specify competencies students must demonstrate to earn a 
degree. On the one hand, it could weil be argued that such requirements should not 
be imposed imtil the colleges have achieved much greater clarity regarding desired 
educational outcomes and much greater confidence regarding assessment 
methodologies. On the other hand, the absence of such graduation requirements 
means that, by default, colleges continue to state degree requirements and award 
credentials on the basis of imits of time allocated for learning, with very little 
emphasis on performance or competency. One significant way colleges are addressing 
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this issue is through the establishment of ''degree warranties" which essentially 
guarantee that the iiistitution will re^ucate or retrain students who do not 
demonstrate competendes defined as requisite to their degree. 

Course and Program Evaluation and Improvement 

The periodic evaluation of courses and programs (especially in vocational and 
technical fields) has been fairly common practice in commimity colleges. In regard 
to these evaluation procedures, a question worth asking is, to what extent is the 
infonnation gathered in these processes actuaUyiised to make improvement^^ (Or 
conversely, are they primarily incestuous paper processes which operate principally 
to preserve the status quo)? A second issue pertains to the apparent emphasis in 
programevaluationprocesseson"academicprogress" and "employment" outcomes 
(including rates for retention, graduation, transfer, job placement, etc.). These 
indicators are clearly important and shoidd be part of assessment efforts; but to date 
(and with a few notable exceptions)., colleges have given fax less attention to the 
direct assessment of student learning outcomes. An outstanding example of work 
in this area is provided by St Petersburg Jimior College, where the faculty has 
developed "end of program assessments" for every instructional program in the 
college. And the results of those student assessments (along with employer surveys, 
advisory conunittee evaluations, and graduation/placement/ transfer rates, etc.) 
are used in evaluating programs. 

Institutional Evaluation 

Givena focus onstudent outcomes, this assessment category encompasses much 
of the survey workimdertakenby commimity colleges, as well as those institutional 
statistics that may be used to describe student academic progress, employment 
outcomes,andoveraUinstitutioiialeffectiveness. There existsomeexceUentex^ 
of such work— models which should be more broadly shared. Survey work and 
manipulation of existing institutional data are in some respects the easiest and least 
expensive assessment devices available — and it is appropriately with this work that 
manyiiistitutionsbeginconstructingtheir assessmentprograrns. Thereare, however, 
many problems that still need to be addressed: poorly designed instruments, poor 
sampling techniques, poor return rates on surveys, difficulty obtaining cooperation 
from four-year institutions, inadequate staffing, and so on. 
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DESIRED STATE: FUTURE DIRECTIONS 
FOR COMMUNITY COLLEGE ASSESSMENT 

The current emphasis on assessment and accountability in higher education 
represents both a significant challenge and a significant opportunity for community 
colleges. The development of a comprehensive assessment programis a conceptually 
and logistically challenging task— one which requires a long-term institutional 
commitment. Many colleges are just beginning their journey on that development 
path; other have gained some experience and are now blazing new trails that others 
may soon follow. In eith ^r case, it can only be helpful to consider in general terms 
where it is we are going. If assessment is to be truly appropriate, truly useful, truly 
matched to the commimity college n:iission — ^what are the issues which colleagues 
on campuses should address? 

Asking the Right Questions 

The crucial cornerstones for an assessment program are laid when the college 
beginsby clearly defining (l)its ownmission, (2) the desired outcomes of community 
college education, and (3) the institution's purposes in xmdertaking assessment. 
Thus, an important first step for each and every institution is to formulate and 
discuss seriously a number of fundamental assessment questions. For example: 

Why are we assessing? 
Who should be involved? 
What exactly should be assessed? 
How will the results be used? 

What methods, procedures, instruments should be used? 

A Systematic Approach 

The more assessment is integrated into the ongoing life of the institution, the 
more effective the program will be. Very important, therefore, are efforts to ensure 
that the results of assessment are used f ystematically in institutional planning and 
budgetiitg, in program, curriculum, and instructional improvement, in designing 
professional development programs, in student monitoring and advisement, and in 
institutional policy development. 
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Focus on Students 



Giventhe valueplacedby conuniinity collegeson "student-centered" approaches, 
one would hope to see increasing emphasis on those forms of assessment which are 
useful in promoting the learning and development of individual students. The 
objective should be not merely assessment 0/ students but assessment students 
and also by them, particularly if the capacity for self-evaluation is seen as useful in 
effective "lifelong learning." Perhaps more than any other type of higher education 
institution, the commimity college must concern itself with the development and 
use of assessment methods that truly help the student to learn. 

For both ethical and legal reasons, it is also crucial that community colleges 
establish effective "safety nets" for students— ways to correct assessment errors, 
systemsforapplyingintelligence to meredata, opportunities forreasoned judgment 
to override statistics. Colleges may well discover that a major result of their 
assessment work (witness the experience of Miami-Dade Community College) is an 
evolution toward more structure and fewer choices for students. Concomitantly, it 
may be necessary to provide additional support services, as large mmibers of 
students struggle to meet higher, more dearly defined expectations. Finally, it is 
incumbent upon community college educators— out of concern for their diverse 
student population- to pursue assessment methods that not only avoid cultural 
and gender bias but even enhance (rather than constrain or deny) culttiral pluralism. 

Improved Teaching and aassrooi a Evaluation 

True to their avowed role as "teaching institutions," community colleges should 
seek routinely to use the results of assessment to make specific improvements in 
teachingandleanung.Fiulher,ahaUmarkoftheassessmentmovementin community 
colleges should be the special effort devoted to improvement of learning assessment 
in the classroom, the result being renewed credibility of certification (grades) at the 
course level. Standardized instruments may be more frequently used in classroom 
evaluation or as a means for periodic validation of faculty assessments. In addition 
(or perhaps as an alternative to standardized tests), colleges may consider using the 
"external examiner" approach to judging student performance. And in the future, 
certification and graduation increasingly will be based on performance (what 
students can do with what they know) rather than on knov/ledge alone or the mere 
accumulation of credit hours. 
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Increased Methodological Sophistication 



As community colleges commit themselves to assessment programs, leaders 
must nurtine concern for and understanding of important methodological and 
technical issues, including appropriate uses of standardized instruments, the 
reliability and validity of both commercial and locally-developed instruments, and 
the broader use of methodologies appropriate to field research. Certainly it is 
recognized that thesotmdestassessmentprograms are those which address miiltipie 
outcomes through multiple methods, in multiple timeframes. Too, commxmity 
colleges should be quick to recognize the importance of building a soxmd and 
comprehensive but "economical" student data base as the cornerstone for an 
effective assessment program. 

Qualitative Emphasis 

In order to avoid "trivializing" the outcomes of commimity college education, it 
iscrudal that attentionbe devoted to the still-embryonic area of qualitative evaluation 
Here lies the opportimity for feculty and others to think seriously about some of 
those important but "ineffable" outcomes of higher education. What, for example, 
does a "responsible citizen" or a "lifelong learner" DO? How do we recognize a 
person who demonstrates "empathy" or "creativity" or "freedom from dogma?" 

Improved Accoimtability 

As their ass^ment work progresses, commimity colleges will hopefully identity, 
acknowledge, and then systematically provide those forms of information about 
student outcomes and institutional effectiveness which policymakers, legislators, 
funding agencies, and the general public have a legitimate need and right to know. 
At the same time, institutional leaders may be communicating to policymakers and 
funding agencies the need for additional resources to support design and 
implementation of assessment programs. And in return for that support, colleges 
will be able to provide demonstizble evidence of improvements made in programs 
and services for students as a result of assessment activities. 

Enhanced Intemal Clim^^te 

Many people have conunented on a particular paradox in the assessment 
movement. One mes^ge going out to the faculty is *at teaching really does matter; 
at the same time, though, faculty also hear the message that their traditional 



classroom evaluation process may be suspect— and that those processes may be 
supplanted by new external evaluation methods. For the society and the educational 
community, it is well past time to eschew faculty-bashing as a coimterproductive 
agenda. Expectations of faculty certainly should be high, as evidenced through 
selection^ orientation, and evaluation procedures; but those expectations should be 
complemented by an emphasis on teaching as the central institutional concern and 
by greatly strengthened professional development programs. 

For their part, faculty will overcome their initial resistance and will likely 
demonstrate some significantly changed attitudes and fresh insights about faculty 
expectationsofstudents,aboutstudentexpectations for their educational 
about the connections between how teachers teach and what students learn, about 
the very definition of what it means to be a faculty member. Assessment may indeed 
produce significant changes in faculty role and identity. As noted by Pat Hutchings 
and Elaine Reuben (1988), faculty are being asked . . to do something against the 
grain, to talk with each other about collective standards and aggregate expectations, 
matters traditionally in the domain of private, professional judgement . . . Lengthy, 
collaborative discussions about teaching and learning, discussions that set out to 
create a shared vision, are not what faculty learn to value in graduate school" (p. 54). 
In a discussion of faculty role in assessment, Paige Cubbison, history professor of 
Miami-Dade Commtmity College, characterizes faculty as "extreme individi .sts, 
resistant to and downright indignant about the idea that all of them would ever do 
anything for the same reason at the same time" (Hu' nhings and Reuben, 1988, p. 54). 
Nonetheless, in institutions where assessment has been taken seriously, faculty 
members report that it is that shared vision, that coherence of effort and sense of 
common purpose, which ultimately make assessment an appealing and rewarding 
enterpri:^. 

GETTING "THERE" FROM "HERE": CONSIDERATIONS 
IN DEVELOPING AN ASSESSMENT PROGRAM 

While there is certainly no "cookbook" for developing the "perfect" assessment 
program^ there are some simple but important things that have been learned from 
the experience of colleges across the country. In most campus environments, there 
are certain conditions which are essential to a successful assessment initiative and 
also certain preparatory steps which will help to make a challengiiig task more 
manageable. 
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Essential Conditions 



Commitment by l op-Lev?l Leaders. Strong, visible commitment, especially by 
the CEO, must be evident. Words are important, but behavior— that is, decisions 
made, actions talcen, resources allocated — is paramount. 

Extensive, Meaningful Involvement In the absence of faculty involvement in 

and ownership of theassessmentprogram,itsimplywillnotresultin improvements 
in curriculun! and teaching. 

Clear and Shared Understanding of Misrdon. The only meaningful basis for 
assessmentis the mission of the institution. Thatmission must be clearly articulated 
and should include an expression of the desired outcomes of commimity college 
education. 



Well-DefinedPurposes for Assessment. Assessment for external accountability 
and assessment for internal improvement need not be mutually exclusive agendas. 
However, in assessment as in any other major institutional initiative, success will 
depend in large part on clarity of purpose. 

Identification of Priorities. Most community colleges have limited resources. 
While it is difficult to make decisions about what things are more important than 

other things,itis far betterinassessmenttodoafew things well thanto falter through 
an tmmanageable agenda. 

Integration of Institutional Processes. The integration of assessment with the 
mainstream of institutional decision-making (e.g... with planning, budgeting, 
curriculum design, faculty development, etc.) is crucial to the usefukiess and long- 
term viability of the effort. 

Determination to Act, Change, Improve. And finally, there is no point in 
expending the requisite time, energy, and money for development of an assessment 
program imless there is a real conunitment to use assessment findings in improving 
programs and services for students. 



Preparation 



To be most effective, assessment programs should be carefully tailored to fit the 
mission, the culture, the curriculum, and the students of each community college. 
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Careful preparation, consisting jiostiy of common-sense measures, can greatiy 
promote the success of that development effort. A few reminders may prove 
helpful: 

1. Inventory and evaluate current assessment activities; use successful 
approaches as building blocks. 

2. Ensure visibility and emphasis for the assessment initiative within the largf c 
context of the organization. 

3. View assessment not as the job of a single person or office or committee but 
as a shared coUegial responsibility. 

4. At the outset, clarify assumptions underlying the assessment program: fcthe 
objective accountability or improvement? Who/what exactiy is being 
evaluated? How will results be used? How will results NOT be used? 

5. Develop or obtain technical expertise. 

6. Establish priorities; then define specific tasks and responsibilities. 
Conceptualize a "grand design/' but be satisfied to "unplementby inaement." 

7. Define incentives for participation, especially by students and faculty, in the 
development and implementation of the program — and also in the use of 
assessment results. 

8. Keep it simple! Insist on usefulness (helping teachers teach, helping students 
learn) as the criterion for selecting assessment approaches. 

9. Commtmicate, communicate, communicate. Often, resistance is heightened 
much more by what people don't know than by what they do know. And 
perhaps the most rewarding aspect of the assessment enterprise is the 
dialogue and debate engendered among colleagues. 

10. Get started. Do not feel that nothing should be done until the research is 
complete and the program design is perfect, for that time will not come. At 
its best, assessment is a continuous learning process, for students, faculty, 
staff, and administrators alike! 

Ina society where a higher educationis intensely scrutinized but also increasingly 
recognized as the cornerstone for developing a compeadve workforce, improving 
the quality of life, and ensuring social justice, it will clearly no longer suffice for 
educators to say "just trust us." In these times, it is even not enough simply to 
perform well. The student, the department or program, and the community college 
must also be able to demonstrate and document that performance. If community 
colleges take that challenge seriously, the achievement of those traditional twin 
goals — excellence and equity — ^may be immeasurably enhanced. 
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DAY 3 



GROUP DISCUSSION AND REPORTS 

Conference participants were once again divided into smaller working groups. 
On their last day, they were asked first to meet in groups as representatives of each 
instituP ^n; those institutional groupings then met with their ACT consultant. The 
task assigned the participants was to davelop a list of factors that they deem critical 
to the success of their assessment project. Each critical success factor has tasks that 
must be accomplished in order to achieve success with the factor. Cov/art briefly 
reviewed the concept ci critical success factors and pointed out the section in a 
demonstration site work book where a page for each critical success factor is 
included. For each critical success factor, participants will ultiTmtebf be asked to 
detennuie the tasks that must be completed, who will be involved in the tasks, how 
long it will take to complete each task, resources required, and an approximate time 
line for accomplishing the tasks. At this point, however, partidpazits were asked 
only to list critical success factors. 

Rofh Group 

Representatives from the following colleges are included in this group: 

Howard Community College 
Midlands Technical College 
Orangebiirg-Calhoim Technical College 
Technical College of the LowCountry 

The first aitical success factor identified by this group is the existence of a mission 
statement that accurately reflects present goals and purposes. If there is a disagreement 
or lack of correspondence, the irission statement should be revised, if possible, or 
brought into alignment with the research and assessment purposes. At a minimum, 
it will be necessary to articulate linkages between the mission statement and the 
project's mission 

The second major factcr critical to the success of the project is to formulate goals 
and obtain faculty /staff involvement. This immediately raises the question: Who should 
be involved? Then: Who should nof be involved? The consensus of the group is that 
no one should be excluded. Since institutional effectiveness and image of the 
institution are a coi :em of everyone, the goal is to be as inclusive as possible. There 
is an appropriateness of involvement for everyone at varying degrees of involvement, 
varying stages, times, and so forth. But, in order to establish ownership, everyone 
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should feel a part of this in volvement Thought should be devoted to how the project 
is presented to others on campus and how to communicate results of the project to 
the foculty. In particular, it is important to determine how best to communicate the 
results of the project to faculty. Is it appropriate to suggest ways that faculty use 
these results? 

Other issues identified include the following: 

a. Identify resources required and compare that list to the list of available 
resourc(5S. Where and when will help be needed? 

b. Establish linkages between Project Cooperation goals and goals already 
existing within the institution and goals relating to external needs. Campus 
groups, operational services on campus, and faculty and staff will all have 
goals, for example, that must be linked to the goals of the project. Further, 
cor*tmunity concerns, other research projects, and grant proposals must be 
linked to the goals of the demonstration site project as well, 

c. It will be important to establish valid administrative tran^g experience for 
anyone directly involved v\dth the project. 

d. It is also critical to maintain spirit, morale, energy, and focus among students, 
faculty/stai^, and especially among Project Cooperation participants. 

What could go wrong? 

• Aiiiition rates in the sample could be higher than expected. Even if that does 
happen, we cannot ignore the important information which could be obtained 
from the entry assessment and surveys along the way about those that we 
lost. 

• Changes in staff— «ome of us may leave and go to another college. That is 
another reason why it is critical to have valid training for administering the 
project. We must be flexible and able to adjusf ihe process. 

Ulmer Group 

Only one institution has Dr. Ulmer as the consultant. Representatives from this 
college had discussed critical success factors the previous evening and were ready 
to present their list. Having already dealt witi; the more general level of critical 
success factors, they felt that the n:^xt stage for thejci would involve the factors listed 
here. Their time in the group session was spent completing the work sheet for one 
critical success factor. First, the list of critical success factors. 

1 . Enhanced faculty support for outcomes assessment. It is important to move 
beyond philosophical commitment to action. 
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2. Renewed administrative support for outcomes assessment. Extend the 
support network. 

3. Establish formal goals and objectives for general education curriculum. 

4. Outline specific outcomes expectations for "exit" competency. 

5. Obtain student commitment and valid participation. 

6. Identify appropriate institutions and secure articulation agreements with 
senior institutions involved in the "predictive" or "transfer" model. 

7. Evaluate and revise the advisor training program. 

8. Identify current strengths and weaknesses of the institution based on long- 
range planning goals that already exist. 

9. Develop a formal procedure to review outcomes data and implementrevisions 
in the curricultun as needed. More systematic procedures are needed. 

10. Develop and/or strengthen the retention program. 

11. Develop and/or strengthen the procedure to review data on a rriculum and 
programs and to assure feedback effectiveness. 

Habley Group 

The Habley group of inslitutions includes the following: 

Chemeketa Commxmity College 
Macomb Community College 
Scottsdale Community College 

Their critical success factors include the following: 

1. Will: Institutional commitment to a long-term assessment program. 

2. A policy statement affirming the institution's support. Some documentation 
of the institutional commitment is needed. Since assessment is not really a 
mission or purpose of the institution, it is perhaps not appropriate to place the 
documentation of the project in a mission statement or policy statement. 
Perhaps a directional docament on assessment is the better idea. (There was 
general discussion of whether this documentation is really useful at this time. 
Depending on the success of the project, assessment may or may not contiunue 
at apy given institution. Without long-term institutionalsupport for this type 
of activity, the effort of implementing the assessment system may bea waste.) 

3. Positive returns on investments— not just monetary, but benefits of a pubKc 
relations and positive image nature are expected too. We caimot be too 
detailed now because at this time we do not know the results of the project. 
Whaf s in it for us? We need to identify the potential. 

4. The design of the project must not impede student flow. There will be 
competition with existing support on an operational system, but it is important 
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that the process of assessing student outcomes not turn out to be something 
that impedes the progress of the student. 

5. Resolving logistical/ operational problems relevant to test administration — 
human, physical, jSnandal, time, and so forth. 

6. Securing people, money, and technical resources. 

7. Clearly communicate the purpose to students, faculty and staff. Students 
need a clear understanding of the purpose. We do not want to "turn off' 
students in the process of assessing them. We must let students know that 
they cannot "fiimk", but that they must do their best when taking the tests. 

8. Provide feedback to participants in the project and to those aware of the 
project. There is a real public relations aspect to this point because it allows 
us to build support for continuing the assessment activities. 

9. Continual attention to addressing barriers and impediments to the project. 
10. SelectthetestmodulesthatactiiaUynmybeinstrumentalinsendingmessages 

to institutions about areas of importance, concem, and concentration. 

Lutz Group 

The final group of institutions has Dr. David Lutz as the consultant for the 
project. The institutions in this group include Metropolitan Commimity Colleges of 
KansasQty,thecoUegesoftheSt.Ix)uis Community CoUegeDistrict,and^ 
State Community College. The following are the ideas of representatives of those 
institutions on what factors are aitical to the success of their outcomes assessment 
project. 

1. Communicate within the college and to external constituencies what the 
assessment is about. 

2. Coordinate the logistics well. 

3. Determine ownership. 

4. Communicate to the publi.c the fact that extensive assessment will be done. 
It is also important to commimicate that course placement and the like 
also be involved in the process. 

5. Assure useful results in a usable form. 

6. Use results prudently and responsibly. 

7. Utilize existing information. 

8. Develop a good research design. 

9. Communicate the results to students. 

10. Get support and commitment from faculty, staff and administration. 

11. Develop and maintain student support. 

12. Proper selecting of students (employ appropriate sampling techniques). 
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LISTING OF PARTICIPANTS 
byAffiUation 



NCSD 

KenAtwater 
Ron Shade 
Barbara Keener 

NOA 

Keith Samuels 

NC31P 
Trudy Bers 
Ix>isWeihe 

Massachusetts Bay Comnjunity 
College 

Roger VanWinkle 
Marge Stewart 
Donna Green 

Piedmont Technical College 
Andy Qmundson 

Chemeketa Commimity College 
Gerard Berger 
Tom Gill 

Dyersburg State Community College 
Buck Tarpley 
Karen Bowyer 

Metropolitan Commimity Colleges 
of Kansas City 
Johnnie McClinton 
Marilyn Donatello 
Alana Timora 
Susan Bourgeois 



Howard Community College 
Martha Matlick 
Barbara Greenf eld 

Macomb Community College 
Charles Eiserunan 
Joseph Sucher 
Don Marcotte 
Ed Lynch 

Midlands Technical College 

Delinda Caxmon 

Dorcas Adams 

Ted McClure 

Sandy Oliver 

Linda Reese 

Technical College of the LowCoimtry 

AnnMdSIutt 
Gafl Quick 
Grace Dennis 
Yvoime Michel 

St Louis Community Colleges 
Betty DuvaU 
Betty Pollard 
James Wheeler 

Orangebxurg-Calhoim Technical 
College 

B?rbara Pickens 
Nancy Coleman 

Scottsdale Community College 
Virginia Stahl 
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ALPHABETICAL LIST OF PARTICIPANTS 



Docas Adams 

Director of Research and Analysis 
Midlands Technical College 
PO Box 2408 
Columbia, SC 29202 

Ken Atwater 
Dean of Student Affeirs 
Catonsville Community College 
800 S. Rolling Road 
Catonsville, MD 21228 

Gerard 1. Berger 

VP for Academic Services 

Chemeketa Commtmity College 

PO Box 14007 

Salem, OR 97309-5008 

Trudy Bers 

Senior Director of Institutional 
Reseaorch, Curriculum, and Planning 
Oakton Community College 
1600 East Golf 
DesPlaines,IL 60016 

Susan Bourgeois 
Learning Assistance Center 
Coordinator 

Metropolitan Community College 
at Penn Valley 
3201 Southwest Trafficway 
Kansas City, MO 64111 

Karen Bowyer 
President 

Dyersburg State Commtmity College 
1516 Nichols Avenue 
Dyersburg, TN 38025-0648 



Delinda Cannon 
Director of Plannmg and 
Administration 
Midlands Technical College 
PO Box 2408 
Coliunbia, SC 29202 

Nancy Coleman 
Director of Assessment 
Orangeburg-Calhoun Technical 
College 

3250 Saint Matthews Road 
Orangeburg, SC 29115 

Grace Dennis 

Dean, Business and General 
Education 

Technical College of the LowCountry 
PO Box 1288 
BeatL^Oit,SC 2990M288 

Marilyn Donatello 
Dean of Campus Services 
Metropolitan Conranmity College 
at Maple Woods 
2601 NE Baixy Road 
Kansas City, MO 64156-1299 

Betty Duvall 
Dean of Instruction 
St. Louis Commxmity College at 
Florissant Valley 
3400 PershaU Road 
St. Louis, MO 63135 
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Charles Eisenman 
Dean of Arts and Sciences 
Macomb Commimity College 
14500 Twelve Mile Road 
Warren, MI 48093-3896 

Tom Gill 

Assistant to the President for 
Institutional Advancement 
Chemeketa Community College 
PO Box 14007 
Salem, OR 9730>5008 

Donna Green 

Associate Dean of Research/ 
Assessment 

Massachusetts Bay Commtmity 
College 

50 Oakland Street 
Wellesley,MD 02181 

Barbara Greenfeld 
Director of Admissions 
Howard Community College 
Little Patiixent Parkway 
Columbia, MD 21044-3197 

Barbara}. Keener 

Dean of Instructional Support 

and Student Services 
Florida Community College at 

Jacksonville 
3939 Roosevelt Blvd. 
Jacksonville, FL 32205 



Don Marcotte 

Macomb Community College 
Consultant 

Department of Planning and 
Evaluation 

Macomb Commimity College 
14500 Twelve Mile Road 
Warren, MI 48093-3896 

Martha Matlick 

Associate Dean, Htonan Services 
Howard Community College 
Little Patuxent Parkway 
Columbia, MD 21044-3197 

Johrmie McClinton 
Assistant Director of Assessment 
and Instructional Services 
Metropolitan Community Colleges 
3200 Broadway 
Kansas City, MO 64411 

TedMcClure 
Dean of Instruction 
Midlands Technical College 
PO Box 2408 
Columbia, SC 29202 

Ann McNutt 
President 

Technical College of the LowCountry 

POBox 1288 

Beaufort, SC 2990M288 

Yvonne Michel 

Director of Research and Planning 
Technical College of the LowCountry 
POBox 1288 
Beaufort, SC 299GM288 
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Sandy Oliver 
Dean of Admissions 
Midlands Technical College 
PO Box 2408 
CoItnnbia,SC 29202 

Andy Osmundson 
Director of Counseling 
Piedmont Technical College 
Emerald Road 
Greenwood, SC 29646 

Barbara Pickens 
Director of Student Services 
Qrangeburg-Calhoim Technical 
College 

3250 Saint Matthews Road 
Orangeburg, SC 29115 

Betty Pollard 

St. Louis Community College 
at Forrest Park 
5600 Oakland Avenue 
Saint Louis, MO 63110 

Betty Pritchard 
Director of Maiketing and 
Communication Services 
Macomb Commimity College 
14500 Twelve Mile Road 
Warren, MI 48093-3896 

Gail Quick 
Dean of Student and 
College Development 
Technical College of the LowCoimtry 
100 S.RibautRoad 
Beaufort, SC 2990M288 



Linda Reese 
Director of Assessment 
Midlands Technical College 
PO Box 2408 
Columbia, SC 29202 

Keith T.Samuels 
VP for Instruction 
Seminole Conununity College 
100 Weldon Blvd. 
Sanford,FL 32773 

Ron Shade 

Dean of Student Affairs 

St. Charles County Community 

College 

102 Compass Point Drive, Suite L 
St. Charles, MO 63301 

Virginia Stahl 

Associate Dean of Instruction 
Scottsdale Commimity College 
9000 East Chaparral 
Scottsdale, AZ 85256-2699 

Marjory Stewart 

Dean of Experimental College 

Massachusetts Bay Conununity 

CoUege 

50 Oakland Street 
Wellesley,MA 02181 

Joseph Sucher 

Director of Planning and Evaluation 
Macomb Community College 
14500 Twelve Mile Road 
Warren, MI 48093-3896 



ERIC 



-67- 



Douglas B. Tarpley 

Director of Institutional Advancement 

Dyersburg State Community College 

1516 Nichols Avenue 

Dyersburg, TN 38025-0648 

Alana Timora 
Assessment Coimselor 
Metropolitan Conamimity College 

at Longview 
500 Longview Road 
Lee's Sunrnut, MO 64063 

Roger Van Winkle 
President 

Massachusetts Ba> Commimity 
College 

50 Oakland Street 
WellesIey,MA 02181 

Lois Weihe 

Dean for Student Services 
Eastern lov^a Community College 
District 
500 BehnontRoad 
Bettendorf, lA 52722 

James Wheeler 
Chair, Dept. of Behavioral 
Sciences 

St. Louis Community College 
at Meramec 
11333 Big Bend Blvd. 
Kirkwood,MO 63122 



ACT 

Susan C. Cowart 
Research Specialist 
National Center for the Advancement 
of Educational Practices 

Wesley R.Habley 
Associate Director 

National Center for the Advancement 
of Educational Practices 

David Lutz 
Director 

College Level Assessment 
and Survey Services 

Frank Potter 

Assistai7\ Director 

Assessment Services 

Southeast Regional Office — Atlanta 

John Roth 
Director 

ASSET and Entry Level Placement 
Exams 

Lovely (Kate) Ulmer 
Assistant Director 
College Level Assessment 
and Survey Senlces 
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DISCUSSION GROUPS 
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